Shaft-in-hole assembly is an essential basic problem in the field of industrial manufacturing. Aiming at its issues of narrow space visual challenges, easy collision damage, low manual assembly efficiency, and other problems, this paper suggests a detection scheme based on binocular vision that uses image fusion to expand the monitoring field-of-view and evaluate assembly parameters through the intersection of shaft and hole. Furthermore, the fusion weight of the binocular data at various positions is optimized according to the reprojection error, which reduces the error by about 10% and ensures the measurement accuracy while realizing fast detection. The experimental findings demonstrate that the suggested method is capable of real-time detection during the assembly process of shafts with small clearance holes and small diameters with a gap of 0.25 mm and an average inaccuracy of roughly 0.015 mm. At the same time, the suggested approach has a strong robustness in various camera installation sites and is not sensitive to the camera's tilt angle.