For characterizing the nanostructure and controlling nano-manufacturing quality,a metrological Atomic Force Microscope(AFM) was designed and constructed in National Institute of Metrology. To trace the displacement to the SI unit, the relative position of sample and AFM probe is measured with homodyne 8-pass interferometers and the surface topology of the sample is measured by AFM at a contact mode. A cube with mirrors is fixed on the probe as the reference mirror of interferometers, so that the relative displacement of probe in the x -y direction to the sample is measured by interferometers. The sample stage is fixed on a corner block with mirrors on three sides and driven by a piezoelectric motion stage. Two interferometers is used to measure the displacement of sample and probe in z direction. The probe tip is positioned in the intersection of the interferometers in 3 directions to minimize the Abbe error. As the phase mixing from the defect of optical element will cause the nonlinear error,a harmonic separation method is introduced to fit the inteferometric signals and to correct the error. The measured results show that the nonlinear error has been reduced to 0.7 nm, which demonstrates this system has better performance.