We present an experimental and theoretical study of self-rotation of optical polarization in a rubidium vapor. The atomic vapor is placed in a magnetic shielding cavity to suppress the Faraday rotation effect. In our experiment, Doppler-free spectroscopy configuration is used, and F=2 -> F'=3 transi