Robot integrated joints are widely used in fields such as medical and collaborative robots, and their friction characteristics are a key factor affecting robot performance. A mechanism model and ensemble learning hybrid driven robot joint friction modeling method is proposed to improve model accuracy. Firstly, taking into account the influencing factors of joint friction characteristics such as speed and load, as well as their periodic fluctuation characteristics, parameterized mechanism models of servo motors and harmonic reducers were established based on prior knowledge and physical analysis to describe the changes in friction characteristics. Then, aiming at the nonlinear residuals caused by linear assumptions and ignoring higher-order terms in mechanism modeling, a residual compensation model modeling method based on eXtreme Gradient Boosting (XGBoost) is proposed. By adopting Boosting ensemble learning strategy, the generalization ability of the residual compensation model is improved. At the same time, Bayesian optimization methods are used to optimize the hyperparameters of the XGBoost model to improve model accuracy and training efficiency. Compared to traditional parameterized mechanism models, the hybrid drive model proposed in this paper has higher accuracy. Comparative experiments with various typical methods such as backpropagation neural networks (BP), support vector machines (SVM), and long short-term memory neural networks (LSTN) show that the residual compensation model based on XGBoost proposed in this paper has stronger feature extraction ability and can better predict strong nonlinear fluctuation friction residuals, effectively improving the accuracy of the overall model.