Defocus dithering technology effectively avoids the non-linear effects introduced by commercial projectors and can achieve high-speed dynamic measurement. However, the binary dithering stripes after defocusing and the standard sinusoidal stripes are not completely close, and there will be a certain deviation in the measurement process. Aiming at the measurement error, this paper proposes an optimization method based on an improved binary ant colony algorithm, which optimizes and improves the binary defocus dithering technology from the phase domain to better improve the measurement accuracy. Through the optimization of small-size binary blocks, the entire optimization process is replaced, and the optimization efficiency is improved. Simulation and experimental results prove that this method can achieve effective three-dimensional measurement under different defocus conditions.