Reinforcement Agent Training: From Q-Learning to Autonomous Control of a Lunar Lander

Python, Reinforcement Learning, PyTorch, Stable-Baselines3, Gymnasium, DQN, PPO, FastAPI, Streamlit

RL Agent Training: From Q-Learning to Autonomous Lunar Lander Piloting

Overview

This project implements a complete progression in reinforcement learning, from theoretical fundamentals to the deployment of a complete autonomous agent. Starting point: the observation → action → reward loop in a simple environment. End point: the autopilot of the "Eagle-1" lunar landing module for AstroDynamics, exposed via an API, visualized in a graphical interface, and monitored by a performance dashboard.

The project applies Reinforcement Learning best practices to build, from scratch to production, a complete chain for training, optimizing, and deploying an intelligent agent.

Objectives

• Understand and manipulate the fundamental observation-action-reward cycle of an RL environment.
• Implement a Q-Learning algorithm from scratch (Q-table) and evaluate its performance.
• Replace the Q-table with a neural network (DQN), manually and then via a reference library.
• Train, optimize, and evaluate an agent capable of solving a continuous/discrete control problem (lunar landing).
• Expose the trained agent via an API, a visualization interface, and a monitoring dashboard.

Skills Demonstrated

Reinforcement Learning Fundamentals

• Getting started with Gymnasium: exploration of observation spaces (continuous, Box) and action spaces (discrete, Discrete).
• Implementation of an agent-environment loop with a random policy, over several episodes, with cumulative reward calculation.

Q-Learning "from scratch"

• Construction and initialization of a Q-table dynamically adapted to the environment dimensions (FrozenLake-v1).
• Implementation of the epsilon-greedy exploration/exploitation strategy with epsilon decay.
• Coding of the Bellman update rule and training over several thousand episodes.
• Rigorous evaluation of the trained agent (pure greedy policy) and calculation of the success rate on a dedicated set of episodes.

Deep Q-Network (DQN)

• Manual implementation in PyTorch of a neural network (DQN) and a ReplayBuffer (based on collections.deque) for experience replay.
• Analysis and handling of a complex DQN training loop: action selection, Q-values calculation, target network, loss calculation, and optimization step.
• Retraining the same problem with Stable-Baselines3 to compare the low-level approach and the tool-based approach (model.learn, evaluate_policy, model saving).

Training and Optimization of a Control Agent (LunarLander)

• Selection of an algorithm adapted to the nature of the action space (PPO for continuous control, DQN for discrete).
• Establishment of a baseline performance before any optimization, to frame the experimentation effort.
• Hyperparameter tuning by isolating one variable at a time (learning rate, gamma, n_steps), monitored via TensorBoard.
• Validation of result robustness (stable average reward, low standard deviation) rather than a score obtained by chance, until exceeding the success threshold of 200 points.

Agent Deployment (lightweight MLOps)

• Development of an API (FastAPI) exposing an inference endpoint accepting a state and returning an action, with all RL logic on the backend side.
• Development of a graphical interface (Streamlit / Gradio) visualizing a game played by the agent.
• Construction of an interactive performance monitoring dashboard (average reward, standard deviation, decisions by context).
• Production of an end-to-end documented notebook and a demonstration video of the autopilot.

Architecture

        Gymnasium Environment
    (CartPole / FrozenLake / LunarLander)
                    │
                    ▼
        Random Policy (baseline)
                    │
                    ▼
     Q-Learning (Q-table, Bellman, ε-greedy)
                    │
                    ▼
   Manual DQN (PyTorch, ReplayBuffer,
        Policy Net / Target Net)
                    │
                    ▼
      DQN / PPO via Stable-Baselines3
      (training, tuning, TensorBoard)
                    │
                    ▼
       Optimized Agent (reward >200)
                    │
        ┌───────────┼───────────┐
        ▼           ▼           ▼
      API        GUI       Dashboard
   (FastAPI)  (Streamlit/    (monitoring of
              Gradio)        performance)

Technical Stack

Reinforcement Learning

• Gymnasium (CartPole-v1, FrozenLake-v1, LunarLander-v3)
• Stable-Baselines3 (DQN, PPO)
• PyTorch (manual DQN implementation)

Monitoring & Optimization

• TensorBoard
• NumPy

Deployment & Visualization

• FastAPI
• Streamlit / Gradio
• Looker Studio

Deliverables

• Documented Google Colab Notebook (.ipynb)
• Demonstration Video (.mp4)
• API and GUI Source Code
• Saved Trained Model (.zip)

Applied Best Practices

• Systematic establishment of a baseline performance before optimization
• Isolation of one hyperparameter at a time to interpret its effect
• Strict separation of the training phase and the evaluation phase (pure greedy policy, no model update)
• Validation of score stability (mean + standard deviation) rather than an isolated result
• Clear separation of responsibilities: RL logic on the backend, visualization on the frontend
• Systematic documentation of each experiment (modified parameters, obtained results)

Results

At the end of this project, the pipeline allows to:

• understand and implement the fundamental mechanisms of RL (Q-learning, DQN) without dependence on a high-level library;
• train a PPO/DQN agent consistently exceeding the set success threshold (average reward > 200 on LunarLander-v3);
• objectively compare a manual implementation and a tool-based implementation (Stable-Baselines3);
• expose the trained agent via an API and a visualization interface usable by a third party;
• continuously monitor the agent's performance via a dedicated dashboard.

Skills Acquired

• Reinforcement Learning Fundamentals (Gymnasium, observation-action-reward cycle)
• Q-Learning and Q-table construction
• Deep Q-Network: manual implementation (PyTorch) and via Stable-Baselines3
• Algorithm choice based on the nature of the action space (PPO vs DQN)
• Hyperparameter tuning and experiment monitoring (TensorBoard)
• Inference API development (FastAPI)
• Construction of visualization interfaces and dashboards (Streamlit, Gradio, Looker Studio)