The research investigation of Response Surface Methodology (RSM) optimization to analyze the impact of load, Algae Oil Methyl Ester (AOME), and CuO copper oxide nano-fuel enhancement on engine performance and exhaust gas emissions, employing both normal and YSZ-coated pistons. The maximum yield observed with 96.32 AOME mix with diesel-biodiesel blends (10, 20, and 30), load variations (10, 50, and 100 kg), and CuO concentrations (50, 75, and 100 ppm) Statistical and experimental data reveal that load and CuO concentrations significantly influence NOx emissions, while AOME blends enhance combustion efficiency, impacting Brake Thermal Efficiency (BTE) and Brake Specific Fuel Consumption (BSFC). In normal and coated piston types, load and CuO concentration considerably affect NOx emissions, although the thermal barrier YSZ coating improves performance at the cost of higher peak flame temperatures and NOx emissions. Experimental results show that with a 60 AOME blend, 75 ppm CuO, and a load of 50 kg, the BTE reached 32 with a BSFC of 0.21 kg/kWh for normal pistons, while YSZ-coated pistons achieved a comparable BTE of 31.1 but with higher NOx emissions (385 ppm compared to 560 ppm for normal pistons). Smoke density reductions of 12.5 were observed with YSZ-coated pistons at optimal conditions. Load, CuO concentration, and AOME blend significantly influenced key performance parameters such as BTE, BSFC, NOx, CO, HC, and smoke density. Surface plots and perturbation diagrams highlighted critical interactions, demonstrating the potential of biodiesel blends, CuO additives, and thermal barrier coatings to improve engine efficiency while minimizing smoke emissions, although at the cost of increased NOx levels. © 2025 Elsevier B.V., All rights reserved.