Magnetic gear motor has the characteristics of high torque density and strong rotor structure. To solve the problem of high cogging torque of external rotor spoke magnetic gear motor, a three-phase 18-slot 28-pole external rotor spoke magnetic gear motor is taken as the research object. The air gap is calculated analytically using the magnetic field modulation theory, and then the mechanism of cogging torque of the external rotor spoke magnetic gear motor is analyzed based on Maxwell tensor method. The magnetic year structure is optimized by adjusting the groove angle and rotor cutting angle. The finite element analysis results show that the third harmonic of the oplimized motor′s back electromotive force is reduced by 53%, the amplitude of cogging torque is reduced by 93.4%, and the output torque ripple is reduced by 73.7%. The finite element analysis results verify the effectiveness of the structural optimization method by adjusting the groove angle and rotor cutting angle.