The continuous growth in traffic demand across metro and core networks is driving operators to adopt cost-effective and sustainable strategies. A promising solution is the deployment of IP-over-WDM (IPoWDM) networks using coherent ZR+ pluggable transceivers connected to high-capacity elastic optical networks. This approach eliminates the need for traditional external transponders, reducing cost, power consumption, and equipment footprint. To enhance sustainability in IPoWDM infrastructures, energy-aware routing, modulation, and spectrum assignment (EA-RMSA) algorithms are crucial for dynamically provisioning connectivity service requests. Traditionally, EA-RMSA has been implemented using heuristics, such as $K$ shortest-path first-fit, aiming to minimize power consumption by accommodating requests on already active devices via a sleep mode strategy. To further improve energy efficiency (i.e., power consumption per throughput), we propose a novel, to the best of our knowledge, deep reinforcement learning-based EA-RMSA solution, referred to as DRL KSP. The trained DRL agent adapts to varying network conditions and learns optimized energy-efficient policies. Performance evaluation under two scenarios with different transceiver configurations and traffic loads shows that DRL EA-KSP achieves power savings up to 7.4% and energy efficiency improvements up to 6.6% compared to heuristic methods. These gains, however, come at the cost of reducing average network throughput by up to 2%, highlighting a trade-off between sustainability and performance. This enables operators to tailor strategies according to their operational goals.