Mode splitters that directly separate modes without changing their orders are highly promising to improve the flexibility of the mode-division multiplexing systems. In this paper, we design a high-performance mode splitter on the silicon-on-insulator platform with a compact footprint of 14 µm× 2.5 µm using an inverse design method based on shape optimization. The fabrication of this mode splitter requires only a single lithography step and exhibits good fabrication tolerances. The experimental results show that the proposed device exhibits state-of-the-art insertion loss (<0.9 dB) and cross talk (<-16 dB) over a broad bandwidth (1500–1600 nm). Furthermore, the shape optimization method used is implemented to design a dual-mode (de)multiplexer, and the experimental results fulfill the design objective, demonstrating the excellent generality of the design method in this paper.