Data centers, the engines of the global Internet, rely on powerful high-speed optical interconnects. In optical fiber communication, classic direct detection captures only the intensity of the optical field, while the coherent detection counterpart utilizes both phase and polarization diversities at the expense of requiring a narrow-linewidth and high-stability local oscillator (LO). Herein, we propose and demonstrate a four-dimensional Jones-space optical field recovery (4-D JSFR) scheme without an LO. The polarization-diverse full-field receiver structure captures information encoded in the intensity and phase of both polarizations, which can be subsequently extracted digitally. To our knowledge, our proposed receiver achieves the highest electrical spectral efficiency among existing direct detection systems and potentially provides similar electrical spectral efficiency as standard intradyne coherent detection systems. The fully recovered optical field extends the transmission distance beyond the limitations imposed by fiber chromatic dispersion. Moreover, the LO-free advantage makes 4-D JSFR suitable for photonic integration, offering a spectrally efficient and cost-effective solution for massively parallel data center interconnects. Our results may contribute to the ongoing developments in the theory of optical field recovery and the potential design considerations for future high-speed optical transceivers.