Through test and complex eigenvalue analysis of DC brush motor used in new energy vehicle, the effect of the friction coefficient between commutator and brush on the vibration mode of the brush system is found, and the mechanism of the motor squeal noise is explained theoretically. When the friction coefficient reaches a critical point, the modes couple, leading to vibration instability and consequently the squeal noise of the brush system. After modification of the structure of the brush spring, the effect of friction coefficient on vibration instability becomes less sensitive and the brush system squealing noise is ameliorated. This study provides guidance and theory to solve the brush squeal noise of DC motors.