In the research of mouse traumatic brain edema, the near infrared spectroscopy effective detection depth is not clear. Aiming at that problem, Monte Carlo method was used to simulate the photon distribution in biological tissue. The photon′s effective detection depth was calculated from the photon distribution. The multi-layer model of mouse brain was obtained by the segmentation of MRI. The advanced value segmentation, narrowband level set were used to segment the scalp, the skull, cerebrospinal fluid, gray and white matter from the whole brain MRI. This segmentation method does not depend on templates. The tMCimg software, which is used to simulate the photons′ optical distribution in complex tissues, was modified to record the photons′ position and the photons′ energy used in the calculation of the effective detection depth. The effective detection depth was discussed with the different center distance of source and detector and radius of source. Effective detection depth in tissue is less than or equal to half of the distance source and detector and it increases with the radius of the source. The mouse brain edema model was built to verify the simulation. The result is important to the design of source and detector probe and the detection of the edema region.