The lattice damage and annealing recovery of N-doped 6H-SiC under neutron irradiation with fluences of 1.72×1019 n/cm2 and 1.67×1020 n/cm2 was investigated by UV-Vis absorption spectroscopy. The results indicate that the significant increase of optical absorption and the decrease of band-gap energy in neutron-irradiated sample is assigned to localized energy level in the forbidden energy band induced by the accumulation of irradiated defects. A continuous strong absorption with many fine structures appeared near the band-edge after irradiation, the origin of which would be attributed to the presence of various different structures of defect clusters and localized amorphous zones. The irradiated samples with two fluences are isochronal annealed from room temperature to 1 600 ℃, the evolution of the absorption spectrum with the annealing temperature shows that the recovery temperature of the lattice damage for the irradiated sample with two fluences is different, but the annealing recovery process shows two same stages with 800 ℃ as the turning temperature.