In the process of approaching a non-cooperative target, it would result in a large amount of calculation using the extracted feature points directly for relative navigation due to the excessive number of feature points. In order to reduce the amount of computation for relative navigation, a subset should be selected from the extracted feature points. Assuming that the extracted feature points were in the same plane of the non-cooperative target, the selection of feature points for non-cooperative target could be transformed as the selection of feature points on the image plane. At the same time, this paper also studied dilution of precision (DOP) for vision-based system and proposed the method that the convex hull of the feature points on the image could be selected as the subset for relative navigation based on the fact that the DOP was inversely proportional to the area of the subset. The simulation shows that the convex hull of feature points can provide high accuracy for relative navigation and reduce the number of feature points effectively. It also shows that the computation time of calculation convex hull is far less than searching the subset with optimal PDOP.