Multiple heterogeneous spectrum users require different Quality of Service(QoS) in cognitive radio networks. A dynamic spectrum allocation method is proposed based on multi-agent reinforcement learning. In order to improve the satisfaction of spectrum users, the proposed method is evaluated by the Quality of Experience(QoE) of spectrum users instead of QoS. Multiple virtual agents are established to simulate spectrum users to learn interactively with environment in a cooperative way, and the optimal spectrum allocation can be obtained by integrating their learning and spectrum decision results. Simulation results show that the proposed method can obtain higher QoE performance of secondary users than those methods based on the traditional reinforcement learning. The probability of collision between spectrum users also can be reduced in the proposed method without any information about the usage rules of primary users and dynamic characteristics of channels.