High-speed single-carrier transmission can be achieved by increasing the modulation format cardinality for higher spectral efficiency. However, ultrahigh-order QAM signals are usually more susceptible to various impairments. Hence, we propose a temporal feature-based memory (TFM) neural network (NN) equalizer to effectively mitigate signals’ impairments in ultrahigh-order QAM. The temporal convolutional network is utilized as a feature extraction layer to significantly improve the performance of the bidirectional long short-term memory network. The TFM-NN equalizer was experimentally validated in a probabilistically shaped polarization-division multiplexed 1024/4096-QAM coherent optical transmission system, and raw spectral efficiencies of 16.190 and 21.188 bit/s/Hz have been achieved at normalized generalized mutual information thresholds.