Utilizing spectrum above 6 GHz (i.e., millimeterwave, mmWave) to enjoy wider bandwidth as compared with that of spectrum below 6 GHz of crowded arrangement has been regarded as an inevitable remedy to sustain high throughput and low latency communications. Suffering from severe signal attenuation, transmission range using mmWave however could be limited, which is a critical concern especially unfavorable for mobile networks. This issue consequently renders beamforming technology as a mandatory function for mmWave communications, but continuous beam sweeping may be inevitable due to the inherent mobility of each user equipment (UE). This operation could be a dominating factor to invoke unacceptable access latency nullifying engineering merits of mmWave. In this paper, we consequently derive foundations of the latency-optimal beam sweeping scheme. For initial access without mobility knowledge, a two-prime modular clock based beam sweeping scheme is proposed to generate beam sweeping sequences, which guarantee bean alignment with the theoretically optimum latency performance. For subsequent access with mobility knowledge, we further derive essential propositions to achieve the ideal performance. The provided scheme consequently offers inevitable principles for the standardization and designs for mmWave based mobile networks.