The mode sorter is the crucial component of the communication systems based on orbital angular momentum (OAM). However, schemes proposed so far can only effectively sort integer OAM (IOAM) modes. Here, we demonstrate the effective sorting of fractional OAM (FOAM) modes by utilizing the coordinate transformation method, which can convert FOAM modes to IOAM modes. The transformed IOAM modes are subsequently sorted using a mode conversion method called topological charge matching. The validation of our scheme is verified by implementing two FOAM sorting processes and corresponding mode purity analyses, both theoretically and experimentally. This new sorting method exhibits great potential for implementing a highly confidential and high-capacity FOAM-based communication and data storage system, which may inspire further applications in both classical and quantum regimes.