Machine learning and ESG integration in portfolio optimization: theory and evidence

We develop an integrated framework for ESG-constrained portfolio management that combines Gram-Schmidt orthogonalization of ESG variables, XGBoost-based return forecasting, constrained mean-variance optimization, Monte Carlo equilibrium analysis with heterogeneous investors, and TOPSIS-based multi-criteria portfolio selection. Using Russell 3000 data (2017–2022), we address three key challenges in sustainable asset management: (i) multicollinearity between ESG and traditional financial factors, resolved through orthogonalization that reduces factor correlations to R2<0.05; (ii) return prediction via machine learning, achieving out-of-sample R2≈8% compared to 3% for linear benchmarks; and (iii) portfolio optimization under binding ESG constraints with non-negativity, solved via quadratic programming. Our main findings are threefold. ESG mandates improve mean-variance efficiency for return targets below 4.56%, reconciling sustainability objectives with fiduciary duties. The equilibrium analysis reveals a negative ESG premium of approximately -45 basis points annually, driven by institutional demand rather than risk compensation. Out-of-sample backtesting confirms that the machine learning-optimized ESG portfolio delivers a Sharpe ratio of 0.72 versus 0.58 for the linear benchmark, with controlled turnover. The TOPSIS selection yields a balanced portfolio with a preference coefficient of 0.68, robust across alternative weighting schemes.