The photonic neural processing unit (PNPU) demonstrates ultrahigh inference speed with low energy consumption, and it has become a promising hardware artificial intelligence (AI) accelerator. However, the nonidealities of the photonic device and the peripheral circuit make the practical application much more complex. Rather than optimizing the photonic device, the architecture, and the algorithm individually, a joint device-architecture-algorithm codesign method is proposed to improve the accuracy, efficiency and robustness of the PNPU. First, a full-flow simulator for the PNPU is developed from the back end simulator to the high-level training framework; Second, the full system architecture and the complete photonic chip design enable the simulator to closely model the real system; Third, the nonidealities of the photonic chip are evaluated for the PNPU design. The average test accuracy exceeds 98%, and the computing power exceeds 100TOPS.