In the process of quantum key distribution (QKD), privacy amplification can eliminate the leaked information of the key data in the QKD process itself and the key information that may be intercept by the eavesdropper, thus ensure the security of the generated quantum key. There are several schemes to realize privacy amplification algorithm based on CPU software. In order to improve the implementation security of this algorithm and the integration level, reduce the power consumption, the implementation of high speed Toeplitz matrix multiplication privacy amplification algorithm based on FPGA was researched. By using matrix block parallel computation, pipeline structure and other accelerated operation methods, the maximum bandwidth of secure key rate of this scheme is up to 20 Mbps when processing 256 kbits input key each time, and up to 5 Mbps when processing 1 Mbits input key each time。In addition, this scheme can adapt to input key length within 1 Mbits, and also adapt to the compress ratio between 0～1, which is conducive to the development of practical high-speed QKD system in the future.