The polarized phase function is one of the important optical parameters, which is very sensitive to the aerosol complex refractive index, particle size and shape. The multi-angle multi-spectral polarization remote sensing is an effective means for obtaining the aerosol polarized phase function. As a ground-based high-accuracy polarization instrument for aerosol remote sensing, the new generation CIMEL dual-polar sun-sky radiometer CE318-DP has been introduced into the worldwide AErosol RObotic NETwork (AERONET). Meanwhile, as the main instrument of the Sun/sky-radiometer Observation NETwork (SONET) with the extension of multi-wavelength polarization measurements, it has accumulated polarization data for many years over observation stations with different aerosol types. However, the retrieval has been based only on the degree of linear polarization or the polarized radiance up to now. Compared to the degree of linear polarization and the polarized radiance, the Stokes parameters Q and U contain information not only on intensity of linear polarization but also on the orientation of polarization. This study introduces an algorithm to retrieve the aerosol polarized phase function based on the Stokes parameters Q and U of skylight from the multi-angle multi-spectral polarization measurements of the CE318-DP. Considering that the Stokes parameter U is changeless with different aerosol properties for the CE318-DP standard polarization observation scenario PPP (Polarized Principal Plane), which is difficult to be utilized, a new ALMP (ALMucantar Polarization) observation scenario is tested to obtain the Stokes parameters Q and U, then to be applied in retrieval of polarized phase function. As to the typical biomass burning and water-soluble aerosols, the results of -P12/P11 in the channels centered at 340 to 1 640 nm were presented and analyzed systematically. Moreover, the applicability of the inversion algorithm in clear and hazy sky conditions was also tested. For the visible and near-infrared channels, the results were in agreement with the truth values not only for the PPP but also for the ALMP geometries. One of the reasons for the obvious deviation of the results in the ultraviolet (UV) bands was also discussed, which was the assumption of the approximately equivalent “ratios of atmospheric single scattering and atmospheric scattering” based on the initial aerosol parameters and the real aerosol parameters could not be satisfied in the UV bands. The inversion model should be improved to be applied to the short-wave channels in further studies. On this basis, some subsequent researches can be engaged in to utilize the features of multi-spectral -P12/P11 to improve the retrieval of aerosol microphysical properties.