The existence of nonlinear friction often affects the performance of servo systems. A composite nonlinear trajectory tracking controller is designed for typical motor servo systems, and the unmeasured speed and unknown disturbance are estimated through a reduced-order extended state observer. In order to eliminate the adverse effects of friction, an adaptive friction compensation mechanism is added into the trajectory tracking controller. Simulation analysis is carried out through MATLAB, and then the control scheme is applied to a DC motor servo system for experimental verification. Simulation and experimental results show that the adaptive compensation scheme can effectively suppress the adverse effects of friction on the system, thereby enabling the controlled system to track the target trajectory quickly and accurately.