When using notch filter to suppress mechanical resonance of servo system, it was necessary to obtain accurate mechanical resonance frequency. In order to detect the mechanical resonance frequency quickly, the adaptive notch filter (ANF) frequency estimation strategy for error signal analysis was proposed. Firstly, the mechanical resonance model of flexible connection servo system was established. Secondly, the time consuming and analysis accuracy were compared between the ANF algorithm and fast Fourier transform (FFT). The numerical and the simulation results show that the proposed method is faster than FFT, and resonance frequency can be identified online quickly. Finally, experimental platform was built and the results of ANF frequency estimation were used as the center frequency of the trap. The suppression of motor speed oscillation was successfully realized, and the effectiveness of the algorithm was verified by experiment.