Optimal Vehicle-to-Grid Control for Supplementary Frequency Regulation Using Deep Reinforcement Learning

The expanding Electric Vehicle (EV) market presents a new opportunity for electric vehicles to deliver a wide range of valuable grid services. Indeed, the emerging Vehicle-to-Grid (V2G) technology with bi-directional flow of power provides the grid with access to mobile energy storage for demand response, frequency regulation and balancing of the local distribution system by storing the surplus of electricity generated from intermittent renewable energy sources in EV batteries during off-peak periods and feeding it back to the grid when needed. This reduces electricity costs at peak hours and can be profitable for customers, network operators and energy retailers. In this paper, an optimal V2G control strategy using Deep Reinforcement Learning (DRL) is proposed to simultaneously maximise the benefits of EV owners and aggregators while fulfilling the driving needs of EV owners. In the proposed DRL-based V2G control strategy, a Deep Deterministic Policy Gradient (DDPG) agent is used for real-time decision-making. The DDPG dynamically adjusts the V2G power scheduling of the EV battery to satisfy the driving demand of EV users and simultaneously perform frequency regulation tasks dispatched by grid operator. The proposed V2G control scheme is tested on a power system consisting of two areas interconnected undergoing frequency deviations. The simulation results have shown that the proposed V2G control can lead to a 22.07% frequency deviation reduction and a 23.15% the Area Control Error (ACE), while satisfying the charging demands of EVs. The proposed V2G strategy has also been compared to other strategies to demonstrate its superior performance. The proposed dispatch strategy is validated on a two-area interconnected power system. In this study, the EV aggregator was assumed to manage 100 EV charging stations each accommodating 500 EVs. The EV parameters such as arriving time, departure time, and the initial and the expected SOC of EV batteries are generated randomly using Monte Carlo method. The simulation results show an improvement in the quality of the frequency regulation while satisfying the charging demands of EVs. The proposed V2G strategy was shown to give a better performance in achieving the expected charging demand for the EVs participating in SFR as compared to other strategies even under large fluctuations in the Area Control Error (ACE) and large negative means values in the Area Regulation Requirement (ARR) signals.