Hybrid DC transmission system often has different types of faults, and the traditional control method takes too long to deal with such faults, so the optimization control method of hybrid DC transmission system based on line-commutated converter (LCC) and modular multilevel converter (MMC) is studied. The topological diagram is drawn according to the system structure characteristics, and the LCC and MMC mathematical models are established. The rectifier side DC voltage is controlled by the triangle and star connection to realize the optimal control. The inverter side MMC is optimized by the voltage-sourced converter (VSC) double-closed-loop controller. By directly charging from the system DC side, the intermediate current transfer process is reduced, and the voltage modulation wave is calculated by the MMC mathematical model to achieve balanced voltage and stable operation of the control system. Simulation results show that the proposed method can control the AC fault of the rectifier station within 5 s. For the DC line unipole fault, the proposed method can quickly respond within 5 s, and the currents of LCC and MMC can be controlled in a stable range. All three currents are effectively controlled, proving that the optimal control of the hybrid DC transmission system is realized and the transmission system faults can be handled quickly.