Since the convergence velocity of mean shift is too slow, fast mean shift for image segmentation is proposed based on conjugate gradient. Conjugate gradient method is characterized by simple, low memory requirements and local and global convergence properties. Moreover, the convergence velocity of conjugate gradient method is between steepest descent method and Newton method. The new algorithm makes use of the properties of conjugate gradient method to improve the convergence velocity of traditional mean shift by interleaved execution of mean shift and conjugate gradient method. Experimental results on synthetic and real images show that new algorithm not only improves the velocity of classical mean shift, but also keeps better segmented result in image segmentation.