In this study, a simple method for measuring the small displacements is presented. When a circularly polarized heterodyne light beam reflected from a mirror is incident into a hemi-spherical prism and is reflected at the base of the prism. Then the reflected light beam passes through an analyzer for interference. With properly chosen azimuth angles of transmission axis of the analyzer, the phase difference between s- andp- polarized light is sensitive to the incident angle near the internal reflection polarization angle. The phase difference can be accurately measured with the heterodyne interferometry. The small displacement of the mirror causes a small variation of incident angle and a phase change. Therefore, substituting the phase difference into special derived equations; the small displacement can be determined. The proposed method has advantages of common-path configuration and heterodyne interferometry.