Edge detection for low-contrast phase objects cannot be performed directly by the spatial difference of intensity distribution. In this work, an all-optical diffractive neural network (DPENet) based on the differential interference contrast principle to detect the edges of phase objects in an all-optical manner is proposed. Edge information is encoded into an interference light field by dual Wollaston prisms without lenses and light-speed processed by the diffractive neural network to obtain the scale-adjustable edges. Simulation results show that DPENet achieves F-scores of 0.9308 (MNIST) and 0.9352 (NIST) and enables real-time edge detection of biological cells, achieving an F-score of 0.7462.