Femtocell is introduced to improve indoor coverage, which is beneficial for both users and operators. But it will also inevitably produce interference management issues in the heterogeneous network which consists of femtocells and macrocells. In this paper, a decentralized Q-learning-based power con