When the Roots vacuum pump drive motor runs under some special environments, the temperature rise of the motor is too high and then causes overburning. In order to solve this problem, the fluid-structure coupling method is used to select the appropriate cooling system to ensure the effective heat dissipation of the motor. Taking a 5 kW motor as the object, the motor simulation models of circumferential spiral channel and axial zigzag channel are established by finite element method. Based on the fluid-structure coupling heat transfer theory, the flow state of the cooling medium in the motor and the motor temperature distribution are analyzed. Taking the flow rate, inlet and outlet pressure difference, and temperature distributions in the motor squirrel cage and winding into consideration, the more suitable cooling water channel structure and size configuration are selected by utilizing the computational fluid dynamics (CFD) software and temperature field analysis. Finally, the temperature rise test of the prototype is carried out, and the test results are in good agreement with the simulation data. The rationality of the cooling system design and the accuracy of the simulation analysis are verified, providing a scientific basis for mass production of the product.