In the field of optoelectronics, certain types of data may be difficult to accurately annotate, such as high-resolution optoelectronic imaging or imaging in certain special spectral ranges. Weakly supervised learning can provide a more reliable approach in these situations. Current popular approaches mainly adopt the classification-based class activation maps (CAM) as initial pseudo labels to solve the task. However, they may fail to estimate the complete object regions, especially in cases of multiple categories existing in one image or tight intersections existing among multiple categories. To solve the problem, we propose a two-branch framework with joint contrastive learning and simulation learning to mine more object regions and produce more complete CAM. Specifically, a contrastive learning branch is designed to learn class-independent activation maps from both foreground and background information, and an original CAM branch is served as supervision to provide accurate discriminative regions. Through simulation learning between the two branches, the enhanced activation maps, which are more complete to cover the objects, are achieved to generate accurate pixel-level pseudo labels. In addition, in order to actively provide important features for contrastive learning, we enhance the backbone in the contrastive learning branch via spatial and channel attention mechanisms. Extensive experiments on the PASCAL VOC 2012 and CUB-200-2011 benchmarks demonstrate that the proposed ICA-Net outperforms many state-of-the-art methods and achieves leading performance.