In order to improve the estimation accuracy of rotor position and speed for sensorless control of the surface-mounted permanent magnet synchronous motor, an improved nonlinear observer is proposed. By reconstructing the mathematical model of the nonlinear observer, the error equation is constructed, and the stability of the error equation is analyzed by using Lyapunov theory. Based on the stability condition, an error step-size optimization method is proposed to realize the self-regulation of the step gain of the nonlinear observer. The simulation and experiment results show that the improved nonlinear observer has good dynamic performance, and the feasibility is verified.