In remote sensing image feature matching, only using feature descriptor similarity measurements results in a large number of outliers. It is important to remove reliably outliers from the initial matching results for improving the accuracy of feature matching and transformation parameter’s estimation. To solve this problem, a simple and effective outlier removal algorithm for remote sensing image feature matching guided by topology is proposed. The potential topological geometric constraints of matching point pairs were completely exploited, and the local and global outlier filtering strategies were presented. The neighborhood consistency of corresponding matching pairs was used, that is, the neighborhood point pairs of the correct matching pairs satisfied the consistency correspondence, and all outlier pairs not meeting the condition were eliminated through a local filtering. Then, based on the hypothesis verification idea of random sampling, global filtering was performed using spatial order constraints and affine area ratio constraints. The local optimization strategy was used to modify the maximum consistent inliers for accurately estimating geometric transformation parameters and reliably removing outliers. Finally, a spatial meshing method was adopted to refine the estimation model and increase the matching pairs to further improve the matching performance of remote sensing images. Compared with other outlier removal algorithms such as NBCS, LPM, LLT, VFC, GMT, SOCBV, and RANSAC, the proposed algorithm is more stable and achieves better performances particularly under complex conditions, including low inlier ratio, severe scale, and viewpoint change.