Researchers investigating autonomous driving systems evaluated an end-to-end reinforcement learning framework powered by Proximal Policy Optimization within the high-fidelity CARLA simulation environment, finding that while policy-gradient continuous control delivers smooth vehicle handling, significant obstacles remain in training tractability and generalization. Traditional autonomous driving systems typically rely on modular pipelines that divide vehicle operation into separate perception, decision-making, and control modules, providing reliable transparency but struggling to adapt flexibly to dynamic, unstructured scenarios. In contrast, end-to-end architectures consolidate the entire driving pipeline into a unified neural model that converts raw sensory inputs directly into real-time driving commands. To assess this learning paradigm, the study examined a foundational Go-To-Point task involving lane-keeping, waypoint navigation, and vehicle velocity regulation in simulated urban streets. Although the resulting end-to-end agent demonstrated smooth trajectory execution and dynamic adaptability during simplified testing, achieving convergence required extensive computational training time and reduced environmental variability, highlighting why traditional modular engineering approaches currently remain more feasible for deployment in practical vehicle systems.

The study’s architectural motivation stems from the fundamental limitations observed in earlier reinforcement learning applications to automated driving. Prior investigations predominantly implemented value-based algorithms such as Deep Q-Networks, which rely on discrete action spaces and often produce abrupt, unnatural steering and throttle adjustments poorly suited to physical vehicular dynamics. Because real-world vehicle operation demands continuous, nuanced modulation of steering angles, acceleration, and braking forces, the authors adopted Proximal Policy Optimization, an on-policy gradient algorithm designed specifically for continuous control domains. Proximal Policy Optimization optimizes a clipped surrogate objective that constrains the magnitude of policy updates during training iterations, mitigating the risk of catastrophic policy destabilization in long-horizon, high-variance driving simulations. This optimization structure enables bounded continuous action outputs, allowing the learning agent to produce fine-grained, stable control signals. By bridging policy-gradient learning and vehicle simulation, the researchers sought to determine whether a unified neural model could effectively master basic navigation without decomposing the task into isolated functional subsystems.

To conduct rigorous and reproducible reinforcement learning experiments, the authors introduced CARLA-GymDrive, an open-source software integration framework that couples the CARLA driving simulator with the standard Gymnasium application programming interface. In autonomous driving research, interfacing simulated physics engines with modern reinforcement learning libraries often presents significant engineering barriers, particularly when configuring custom action and observation spaces. CARLA-GymDrive resolves these technical hurdles by establishing a standardized environment interface compatible with established reinforcement learning libraries, specifically utilizing the Stable-Baselines3 implementation of the Proximal Policy Optimization algorithm. The software framework provides real-time sensor visualization, graphical telemetry inspection, and configurable reward functions that allow researchers to calibrate experimental parameters such as route progress, safety margins, and time-to-destination. Furthermore, the integration supports both discrete and continuous action representations, enabling low-latency communication between the learning agent and the CARLA physics engine while abstracting the underlying simulator configuration to lower the technical barrier for automated driving experimentation.

The experimental setup utilized a simulated Tesla Model 3 vehicle operating within the Town01 map in CARLA, an urban layout containing a combination of straight roads, intersections, and turns. The ego vehicle was equipped with an RGB camera mounted directly on the windshield, providing forward-facing visual perspectives matching a human driver’s viewpoint, alongside a Global Navigation Satellite System sensor for position tracking, a collision sensor to record impacts, and a lane invasion sensor to monitor lane boundary adherence. Rather than feeding raw high-dimensional localization coordinates that increase computational overhead, the researchers designed a compact observation space combining an RGB camera image of dimensions 360 by 640 pixels across three color channels with three scalar features: distance to the final target, distance to the nearest waypoint, and the ego vehicle’s current speed. In the neural network architecture, image frames were processed through a custom Convolutional Neural Network, while scalar variables passed through a Multi-Layer Perceptron projected to a 128-element feature vector to prevent richer visual inputs from overshadowing numerical navigation signals, before both streams merged into a final decision layer.

Vehicle control commands were structured through a streamlined continuous action space bounded between negative one and positive one across two distinct control dimensions. To reduce policy complexity, the researchers designed an innovative fused throttle and braking channel where positive values dictate acceleration intensity and negative values correspond to braking force, departing from CARLA’s default separation of throttle and brake pedals. A separate continuous dimension governed steering commands, with negative one representing a full left turn and positive one representing a full right turn. To guide the agent through typical 30-second episodes spanning approximately 430 simulation time steps, the team formulated a multi-component reward function balancing velocity regulation, target proximity, waypoint adherence, and collision avoidance. The agent received positive rewards for maintaining target travel speeds without prolonged standstill, incremental distance-based rewards that increased as the vehicle neared its goal, fixed bonuses for passing five designated route checkpoints, and a substantial terminal reward of 100 points upon successfully reaching the target, while collisions triggered severe numerical penalties.

Training the end-to-end model was executed across a two-phase curriculum designed to balance environmental generalization against learning stability. In the initial phase, the ego vehicle was placed in diverse urban scenarios characterized by changing weather conditions, including clear skies, heavy rain, fog, and low-light environments, alongside geometric variations such as intersections, narrow lanes, and sharp turns. However, this high environmental variability caused significant training instability; the agent exhibited prolonged periods of inactivity, experienced high variance in policy updates, and failed to complete any evaluation routes within the allocated training timeframe. In response to these convergence difficulties, the authors transitioned to a simplified second phase that removed dynamic traffic, eliminated pedestrians, and fixed the weather conditions and urban layout to a consistent map setting. Across this simplified regimen, the vehicle was trained for 120,000 episodes, mirroring benchmark durations used in related literature to establish a standardized baseline for policy convergence.

To establish an empirical baseline against which the end-to-end model could be evaluated, the researchers constructed a traditional modular autonomous driving pipeline tested under identical environmental conditions in Town01. The modular architecture decomposed driving operations into three separate components: perception, rule-based decision-making, and execution control. The perception component bypassed deep learning, directly ingesting ground-truth simulation data from CARLA, including vehicle velocity, heading, and global waypoint coordinates, mimicking a high-level perception layer. The rule-based decision-making module evaluated upcoming waypoints along the planned route and determined heading directions using geometric calculations, intentionally omitting dynamic obstacle avoidance to match the static conditions of Phase 2. For execution, vehicle controls were handled using the native carla.VehicleControl interface for continuous control tests and the CARLA Ackermann Control module for discrete steering abstractions. This segmented modular architecture provided high interpretability and deterministic stability, enabling the researchers to directly contrast an engineered pipeline with the end-to-end model’s holistic learning approach.

Quantitative evaluation of the trained end-to-end agent demonstrated marked performance differences between the two training regimens, confirming the functional viability of the continuous control policy under structured conditions. While Phase 1 models failed to complete test scenarios due to early stagnation, Phase 2 agents achieved substantial improvements, successfully completing seven deterministic test scenarios and reliably reaching four out of five route checkpoints. Normalized reward values averaged approximately 0.320, an intermediate figure directly explained by the mathematical structure of the reward function, which reserved a large discrete bonus of 100 points for full destination completion. The agent produced smooth steering trajectories and stable acceleration transitions, avoiding the erratic oscillation typical of discrete action models. However, attaining this level of performance required approximately 480 hours of computational training on an Intel Core i7 processor and an Nvidia RTX 3060 graphics processing unit, underscoring the massive computational resources required by end-to-end reinforcement learning architectures even within simplified simulation domains.

Despite successful waypoint tracking in Phase 2, the empirical findings revealed critical trade-offs regarding generalization and experimental scope. The environmental simplifications required to stabilize policy convergence—specifically the elimination of pedestrian traffic, moving vehicles, and adverse weather—induced localized policy overfitting, leaving the trained neural agent ill-equipped to handle novel or complex urban interactions. The authors highlighted several threats to validity inherent to the simulation setup, noting that deterministic CARLA environments exclude spontaneous pedestrian crossings, erratic human motoring behaviors, sensor noise, and environmental transitions found on public roadways. Additionally, the experimental scope evaluated only single-episode runs on straight road segments in Phase 2, omitting multi-episode endurance testing necessary to measure long-term controller drift, cumulative errors, and behavioral fatigue. Practical automotive criteria such as passenger ride comfort, braking frequency, and energy consumption metrics were also unmeasured, though qualitative observations confirmed that the lightweight convolutional network executed in real time on consumer-grade hardware without noticeable frame latency.

In analyzing the broader implications of their findings, the authors concluded that while end-to-end reinforcement learning offers architectural elegance by eliminating inter-module communication and error propagation, traditional modular systems currently remain superior for real-world automated vehicle deployment. Modular systems offer clear interpretability, predictable failure isolation, and manageable component-level pretraining, whereas end-to-end architectures behave as opaque models that require prohibitive training durations and exhibit limited cross-domain transferability. The research demonstrates that policy-gradient algorithms like Proximal Policy Optimization combined with unified action spaces can produce smooth vehicle control, but closing the gap toward real-world autonomous driving requires substantial architectural refinements. Future research directions proposed by the authors include integrating curriculum learning, domain randomization across multiple urban maps, semantic perception representations, and hierarchical or multi-agent reinforcement learning algorithms to expand vehicle capabilities beyond basic point-to-point navigation while profiling per-frame inference latency under realistic operating constraints.

Subject of Research: End-to-end reinforcement learning architectures for autonomous driving

Article Title: Evaluating end-to-end autonomous driving architectures: a proximal policy optimization approach in simulated environments

Article References: Morgado, Â., Ota, K., Dong, M., & Pombo, N. (2025). Evaluating end-to-end autonomous driving architectures: a proximal policy optimization approach in simulated environments. Autonomous Intelligent Systems, 5(1), Article 14. https://doi.org/10.1007/s43684-025-00102-3

Image Credits: AI Generated

DOI: 10.1007/s43684-025-00102-3

Keywords: autonomous driving, reinforcement learning, proximal policy optimization, CARLA simulator, CARLA-GymDrive, continuous control, modular architecture, vehicle simulation, Evaluating, end-to-end, autonomous, driving