Aiming at the unknown clutter intensity existed in Probability Hypothesis Density (PHD) multi-target tracking algorithm, a tracking algorithm based on Entropy Penalized Expectation Maximization (EPEM) of unknown clutter estimation is proposed, called EPEM-PHD. The clutter intensity is modeled by finite mixture model. The entropy penalized factor is applied on mixed weight and the missing parameter. By adjusting the adaptive coefficient, the extinction of components with low weight is accelerated, thus can decrease the times of iterations. And the algorithm is not sensitive to initial parameters. Simulation results show that: the algorithm has the advantages of high precision and stable tracking, which improves the performance of PHD filter in multi-target tracking.