A method based on the iterative inversion estimation of transport mean free path and the minimum source-detector distance determination was proposed. The method utilizes the idea of iterative estimation and adaptively changes the minimum source-detector distance to improve the inversion accuracy of optical property parameters. The study of 29 sets simulation data indicates that the iterative inversion is not sensitive to the initial value of the transport mean free path, and has a higher robustness. In a certain extent, the inversion accuracy of the two parameters can be improved. Under the condition of no noise, the average relative error of the absorption coefficient is 7.17%, and the average relative error of the reduced scattering coefficient is 5.73%. Compared with the best result of traditional empirical method for a given minimum source-detector distance, average relative error decreases 1.73% for the absorption coefficient and 1.14% for the reduced scattering coefficient. In the case of 40~80 dB for signal-to-noise ratio, the method can still obtain higher inversion accuracy of optical property parameters. The error ranges of the absorption coefficient and the reduced scattering coefficient are 8.46%~10.05% and 6.79%~8.76%, respectively.