{"id":"e6e148f3-d4f3-4cdd-9826-22d381cc63ac","authorId":"e1bbfa18-4766-4421-b638-f9c14a6b81b6","title":"Optimizing RL Agents with Exit Rules in RLXBT","slug":"optimizing-rl-agents-with-exit-rules-in-rlxbt","excerpt":"\nIn this article, we explore how using strict Exit Rules affects the training and performance of Reinforcement Learning (RL) agents in the cryptocurrency market.","content":"## Introduction\n\nOne of the main challenges in training RL agents for trading is the \"noisy\" reward signal. It's difficult for an agent to understand whether a trade was successful due to its entry or simply because of a lucky turn of events. Exit rules allow us to separate the **entry** logic (which the agent learns) from the **risk management** logic (which is strictly defined).\n\n## Exit Rules Configurations\n\nIn our experiment, we compared three approaches:\n\n1.  **No Rules (Baseline)**: The agent decides when to close a position.\n2.  **Conservative**: Strict 2% stop-loss, 3% take-profit, and a maximum holding time of 48 hours.\n3.  **Aggressive**: 5% stop-loss, quick 1.5% take-profit, and holding for no more than 12 hours.\n\n---\n\n## Experimental Results\n\nData: BTCUSDT, 1-hour timeframe (2020-2025).\n\n### Training Summary Table (PPO Agent)\n\n| Configuration                   |   Return    | Sharpe Ratio | Max Drawdown | Total Trades |\n| :------------------------------ | :---------: | :----------: | :----------: | :----------: |\n| **No Rules**                    |   -5.14%    |   -0.0075    |    14.71%    |     744      |\n| **Conservative (2% SL, 3% TP)** |   -3.27%    |   -0.1234    |    4.46%     |      17      |\n| **Aggressive (5% SL, 1.5% TP)** | **+34.58%** |  **0.0407**  |    20.15%    |     1242     |\n\n### Exit Reason Analysis (for the best strategy)\n\nFor the aggressive strategy, which showed the best result, the distribution of position closing reasons is as follows:\n\n- **Signal (Agent Signal):** 88.6%\n- **MaxBarsReached (Timeout):** 6.5%\n- **MinProfitReached (Take-Profit):** 4.7%\n- **MaxDrawdown (Stop-Loss):** 0.2%\n\n**Conclusion:** The agent learned to effectively use short market impulses, while the exit rules provided a safety net during prolonged movements or sharp drawdowns.\n\n---\n\n## Full Example Code\n\nBelow is the full script code to reproduce the results. To run it, you will need the `rlxbt` and `stable-baselines3` libraries installed.\n\n```python\n#!/usr/bin/env python3\n\"\"\"\nRLX RL Environment Demo with Exit Rules\n\nThis demo shows how to:\n1. Configure RlxEnv with custom exit rules\n2. Train an RL agent (PPO) with risk management\n3. Compare performance with/without exit rules\n4. Generate detailed metrics and reports\n\nExit Rules Features:\n- hold_bars: Maximum bars to hold a position\n- max_drawdown_percent: Force exit if position drawdown exceeds threshold\n- min_profit_percent: Take profit when minimum target reached\n- exit_at_night: Close positions during night hours\n- max_hold_minutes: Time-based exit\n\nLICENSING:\n- Set RLX_LICENSE_KEY environment variable or pass license_key parameter to RlxEnv\n- For development builds (--features offline_license), license is not required\n- Get your license at https://rlxbt.com/pricing\n\"\"\"\n\nimport sys\nimport os\nimport time\nimport numpy as np\nimport pandas as pd\nfrom dotenv import load_dotenv\n\n# Load environment variables from .env file\nload_dotenv()\n\n# Add project root to path\nproject_root = os.path.dirname(\n    os.path.dirname(os.path.dirname(os.path.abspath(__file__)))\n)\nsys.path.insert(0, project_root)\n\ntry:\n    from stable_baselines3 import PPO\n    from stable_baselines3.common.vec_env import DummyVecEnv\n    from stable_baselines3.common.callbacks import BaseCallback\n\n    HAS_SB3 = True\nexcept ImportError:\n    HAS_SB3 = False\n    print(\"⚠️  stable_baselines3 not installed. Running simplified demo.\")\n\ntry:\n    from rlxbt import rlx, load_data, RlxEnv\nexcept ImportError:\n    print(\"❌ Failed to import RLX. Please run 'maturin develop' first.\")\n    sys.exit(1)\n\n\nclass RewardCallback(BaseCallback):\n    \"\"\"Callback to track training progress.\"\"\"\n\n    def __init__(self, verbose=0):\n        super().__init__(verbose)\n        self.episode_rewards = []\n        self.episode_count = 0\n\n    def _on_step(self) -> bool:\n        if self.locals.get(\"dones\", [False])[0]:\n            self.episode_count += 1\n            if self.episode_count % 10 == 0:\n                info = self.locals.get(\"infos\", [{}])[0]\n                portfolio = info.get(\"portfolio_value\", 100000)\n                ret = (portfolio - 100000) / 100000 * 100\n                print(\n                    f\"  Episode {self.episode_count}: Portfolio ${portfolio:,.0f} ({ret:+.2f}%)\"\n                )\n        return True\n\n\ndef run_episode_manual(env, strategy=\"random\"):\n    \"\"\"Run single episode with manual strategy (no RL library needed).\"\"\"\n    obs, _ = env.reset()\n    done = False\n    total_reward = 0\n    actions_taken = []\n\n    while not done:\n        if strategy == \"random\":\n            action = np.random.choice([0, 1, 2])\n        elif strategy == \"always_long\":\n            action = 1\n        elif strategy == \"always_short\":\n            action = 2\n        else:  # hold\n            action = 0\n\n        obs, reward, done, truncated, info = env.step(action)\n        total_reward += reward\n        actions_taken.append(action)\n\n    return total_reward, info, actions_taken\n\n\ndef main():\n    print(\"=\" * 70)\n    print(\"🤖 RLX RL ENVIRONMENT WITH EXIT RULES DEMO\")\n    print(\"=\" * 70)\n\n    # Check for license key\n    license_key = os.environ.get(\"RLX_LICENSE_KEY\")\n    if license_key:\n        print(f\"🔑 Using license key: {license_key[:20]}...\")\n    else:\n        print(\"ℹ️  No RLX_LICENSE_KEY set (OK for development builds)\")\n        print('   For production: export RLX_LICENSE_KEY=\"rlx_pro_...\"')\n\n    # =========================================================================\n    # 1. LOAD DATA\n    # =========================================================================\n    data_path = os.path.join(\n        project_root, \"data\", \"BTCUSDT_1h_2020-12-12_2025-12-11.csv\"\n    )\n\n    if not os.path.exists(data_path):\n        print(f\"❌ Data file not found: {data_path}\")\n        return\n\n    print(f\"\\n📊 Loading data from: {os.path.basename(data_path)}\")\n    data = load_data(data_path)\n    print(f\"   Total bars: {len(data):,}\")\n    print(f\"   Date range: {data['timestamp'].min()} to {data['timestamp'].max()}\")\n\n    # Split data\n    train_size = int(len(data) * 0.7)\n    val_size = int(len(data) * 0.15)\n\n    train_data = data.iloc[:train_size].reset_index(drop=True)\n    val_data = data.iloc[train_size : train_size + val_size].reset_index(drop=True)\n    test_data = data.iloc[train_size + val_size :].reset_index(drop=True)\n\n    print(f\"\\n📈 Data Split:\")\n    print(f\"   Train: {len(train_data):,} bars (70%)\")\n    print(f\"   Valid: {len(val_data):,} bars (15%)\")\n    print(f\"   Test:  {len(test_data):,} bars (15%)\")\n\n    # =========================================================================\n    # 2. DEFINE EXIT RULES CONFIGURATIONS\n    # =========================================================================\n    print(\"\\n\" + \"=\" * 70)\n    print(\"⚙️  EXIT RULES CONFIGURATIONS\")\n    print(\"=\" * 70)\n\n    # Configuration 1: No Exit Rules (baseline)\n    no_rules = None\n\n    # Configuration 2: Conservative Risk Management\n    conservative_rules = {\n        \"hold_bars\": 48,  # Max 48 hours (2 days)\n        \"max_drawdown_percent\": 2.0,  # Stop loss at 2% drawdown\n        \"min_profit_percent\": 3.0,  # Take profit at 3%\n    }\n\n    # Configuration 3: Aggressive Day Trading\n    aggressive_rules = {\n        \"hold_bars\": 12,  # Max 12 hours\n        \"max_drawdown_percent\": 5.0,  # Allow 5% drawdown\n        \"min_profit_percent\": 1.5,  # Quick profit taking at 1.5%\n    }\n\n    # Configuration 4: Session-Based (Night Exit)\n    session_rules = {\n        \"hold_bars\": 24,  # Max 24 hours\n        \"exit_at_night\": True,  # Close before night\n        \"night_start_hour\": 22,  # Night starts at 22:00 UTC\n        \"night_end_hour\": 6,  # Night ends at 06:00 UTC\n        \"max_drawdown_percent\": 3.0,\n    }\n\n    configs = [\n        (\"No Rules (Baseline)\", no_rules),\n        (\"Conservative\", conservative_rules),\n        (\"Aggressive\", aggressive_rules),\n        (\"Session-Based\", session_rules),\n    ]\n\n    for name, rules in configs:\n        print(f\"\\n📋 {name}:\")\n        if rules:\n            for k, v in rules.items():\n                print(f\"   {k}: {v}\")\n        else:\n            print(\"   No exit rules applied\")\n\n    # =========================================================================\n    # 3. TEST RANDOM AGENT WITH DIFFERENT CONFIGS\n    # =========================================================================\n    print(\"\\n\" + \"=\" * 70)\n    print(\"🎲 RANDOM AGENT COMPARISON (baseline)\")\n    print(\"=\" * 70)\n\n    random_results = []\n\n    for config_name, exit_rules in configs:\n        # License key is automatically read from RLX_LICENSE_KEY environment variable\n        env = RlxEnv(\n            data=test_data,\n            initial_capital=100000.0,\n            window_size=20,\n            exit_rules=exit_rules,\n        )\n\n        # Run 5 episodes with random actions\n        returns = []\n        trades_list = []\n        for _ in range(5):\n            _, info, _ = run_episode_manual(env, strategy=\"random\")\n            returns.append(info.get(\"total_return\", 0) * 100)\n            trades_list.append(int(info.get(\"total_trades\", 0)))\n\n        avg_return = np.mean(returns)\n        avg_trades = np.mean(trades_list)\n\n        random_results.append(\n            {\n                \"config\": config_name,\n                \"avg_return\": avg_return,\n                \"avg_trades\": avg_trades,\n                \"std_return\": np.std(returns),\n            }\n        )\n\n        print(f\"\\n{config_name}:\")\n        print(f\"   Avg Return: {avg_return:+.2f}% (±{np.std(returns):.2f}%)\")\n        print(f\"   Avg Trades: {avg_trades:.0f}\")\n\n    # =========================================================================\n    # 4. TRAIN RL AGENTS FOR EACH CONFIG (if stable_baselines3 available)\n    # =========================================================================\n    if HAS_SB3:\n        print(\"\\n\" + \"=\" * 70)\n        print(\"🧠 RL AGENT TRAINING (PPO) - Training separate agent per config\")\n        print(\"=\" * 70)\n\n        # Training configurations (only train with rules that make sense)\n        train_configs = [\n            (\"No Rules\", no_rules),\n            (\"Conservative (2% SL, 3% TP)\", conservative_rules),\n            (\"Aggressive (5% SL, 1.5% TP)\", aggressive_rules),\n        ]\n\n        eval_results = []\n        trained_models = {}\n\n        for config_name, exit_rules in train_configs:\n            print(f\"\\n🏋️ Training PPO agent with: {config_name}\")\n            if exit_rules:\n                print(f\"   Exit Rules: {exit_rules}\")\n\n            # Create training environment\n            # Use lambda with default argument to capture exit_rules correctly\n            train_env = DummyVecEnv(\n                [\n                    lambda er=exit_rules: RlxEnv(\n                        data=train_data,\n                        initial_capital=100000.0,\n                        window_size=32,  # Optimized window size\n                        exit_rules=er,\n                    )\n                ]\n            )\n\n            # Create PPO model with optimized hyperparameters\n            model = PPO(\n                \"MlpPolicy\",\n                train_env,\n                verbose=0,\n                learning_rate=3e-4,\n                n_steps=1024,\n                batch_size=64,\n                n_epochs=10,\n                gamma=0.99,\n                ent_coef=0.02,  # Higher entropy for exploration\n            )\n\n            # Training\n            print(f\"   Training for 100,000 timesteps...\")\n            start_time = time.time()\n            model.learn(total_timesteps=100_000)\n            train_time = time.time() - start_time\n            print(f\"   Training completed in {train_time:.1f}s\")\n\n            trained_models[config_name] = model\n\n            # =====================================================================\n            # 5. EVALUATE ON TEST SET\n            # =====================================================================\n            test_env = RlxEnv(\n                data=test_data,\n                initial_capital=100000.0,\n                window_size=32,\n                exit_rules=exit_rules,\n            )\n\n            obs, _ = test_env.reset()\n            done = False\n            actions = {0: 0, 1: 0, 2: 0}\n\n            while not done:\n                action, _ = model.predict(obs, deterministic=True)\n                action = int(action)\n                actions[action] += 1\n                obs, reward, done, truncated, info = test_env.step(action)\n\n            total_actions = sum(actions.values())\n\n            result = {\n                \"config\": config_name,\n                \"total_return\": info.get(\"total_return\", 0) * 100,\n                \"sharpe_ratio\": info.get(\"sharpe_ratio\", 0),\n                \"max_drawdown\": info.get(\"max_drawdown\", 0) * 100,\n                \"total_trades\": int(info.get(\"total_trades\", 0)),\n                \"win_rate\": info.get(\"win_rate\", 0) * 100\n                if info.get(\"win_rate\")\n                else 0,\n                \"portfolio_value\": info.get(\"portfolio_value\", 100000),\n                \"hold_pct\": actions[0] / total_actions * 100,\n                \"long_pct\": actions[1] / total_actions * 100,\n                \"short_pct\": actions[2] / total_actions * 100,\n                \"train_time\": train_time,\n            }\n            eval_results.append(result)\n\n            print(f\"\\n   📊 Test Results:\")\n            print(f\"   Total Return:    {result['total_return']:+.2f}%\")\n            print(f\"   Sharpe Ratio:    {result['sharpe_ratio']:.4f}\")\n            print(f\"   Max Drawdown:    {result['max_drawdown']:.2f}%\")\n            print(f\"   Total Trades:    {result['total_trades']}\")\n            print(\n                f\"   Actions: Hold={actions[0]} ({result['hold_pct']:.1f}%), \"\n                f\"Long={actions[1]} ({result['long_pct']:.1f}%), \"\n                f\"Short={actions[2]} ({result['short_pct']:.1f}%)\"\n            )\n\n        # =====================================================================\n        # 6. SUMMARY TABLE\n        # =====================================================================\n        print(\"\\n\" + \"=\" * 70)\n        print(\"📊 RESULTS SUMMARY - Each agent trained with its own config\")\n        print(\"=\" * 70)\n\n        print(\"\\n┌\" + \"─\" * 78 + \"┐\")\n        print(\n            f\"│ {'Config':<32} {'Return':>10} {'Sharpe':>10} {'Drawdown':>10} {'Trades':>8} │\"\n        )\n        print(\"├\" + \"─\" * 78 + \"┤\")\n        for r in eval_results:\n            print(\n                f\"│ {r['config']:<32} {r['total_return']:>+9.2f}% {r['sharpe_ratio']:>10.4f} \"\n                f\"{r['max_drawdown']:>9.2f}% {r['total_trades']:>8} │\"\n            )\n        print(\"└\" + \"─\" * 78 + \"┘\")\n\n        # Best config\n        best = max(eval_results, key=lambda x: x[\"sharpe_ratio\"])\n        print(f\"\\n🏆 Best Configuration: {best['config']}\")\n        print(f\"   Sharpe Ratio: {best['sharpe_ratio']:.4f}\")\n        print(f\"   Total Return: {best['total_return']:+.2f}%\")\n        print(f\"   Max Drawdown: {best['max_drawdown']:.2f}%\")\n\n        # =====================================================================\n        # 7. EXIT STATISTICS (using best model)\n        # =====================================================================\n        print(\"\\n\" + \"=\" * 70)\n        print(f\"📈 EXIT REASONS ANALYSIS ({best['config']})\")\n        print(\"=\" * 70)\n\n        # Use the best performing model for analysis\n        best_model = trained_models.get(best[\"config\"])\n        best_rules = None\n        for name, rules in train_configs:\n            if name == best[\"config\"]:\n                best_rules = rules\n                break\n\n        if best_model and best_rules:\n            analysis_env = RlxEnv(\n                data=test_data,\n                initial_capital=100000.0,\n                window_size=32,\n                exit_rules=best_rules,\n            )\n\n            obs, _ = analysis_env.reset()\n            done = False\n\n            while not done:\n                action, _ = best_model.predict(obs, deterministic=True)\n                obs, reward, done, truncated, info = analysis_env.step(int(action))\n\n            # Get backtest result for exit statistics\n            try:\n                backtest_result = analysis_env.get_backtest_result()\n\n                # Count exit reasons\n                exit_reasons = {}\n                for trade in backtest_result.trades:\n                    reason = (\n                        str(trade.exit_reason)\n                        if hasattr(trade, \"exit_reason\")\n                        else \"Unknown\"\n                    )\n                    exit_reasons[reason] = exit_reasons.get(reason, 0) + 1\n\n                if exit_reasons:\n                    print(\"\\nExit Reason Distribution:\")\n                    total_exits = sum(exit_reasons.values())\n                    for reason, count in sorted(\n                        exit_reasons.items(), key=lambda x: -x[1]\n                    ):\n                        pct = count / total_exits * 100\n                        print(f\"   {reason:<30} {count:>5} ({pct:>5.1f}%)\")\n\n            except Exception as e:\n                print(f\"   Could not get exit statistics: {e}\")\n\n    else:\n        print(\"\\n⚠️  Skipping RL training (stable_baselines3 not installed)\")\n        print(\"   Install with: pip install stable-baselines3 shimmy gymnasium\")\n\n    # =========================================================================\n    # 8. FINAL NOTES\n    # =========================================================================\n    print(\"\\n\" + \"=\" * 70)\n    print(\"📝 KEY TAKEAWAYS\")\n    print(\"=\" * 70)\n    print(\"\"\"\n1. EXIT RULES IMPACT:\n   - Conservative rules (2% SL, 3% TP) reduce risk but may limit upside\n   - Aggressive rules allow bigger swings, higher variance\n   - Session-based rules useful for avoiding overnight gaps\n\n2. RL + EXIT RULES SYNERGY:\n   - RL agent learns WHEN to enter (signal timing)\n   - Exit rules handle risk management (HOW to exit)\n   - This separation allows cleaner learning signal\n\n3. CONFIGURATION RECOMMENDATIONS:\n   - Day Trading: aggressive_rules with short hold_bars\n   - Swing Trading: conservative_rules with longer hold_bars\n   - 24/7 Markets (Crypto): no night exit needed\n   - Traditional Markets: session_rules with night exit\n\n4. HYPERPARAMETER TUNING:\n   - hold_bars: Match your trading timeframe\n   - max_drawdown_percent: Set based on risk tolerance\n   - min_profit_percent: Balance between taking profits and letting winners run\n\"\"\")\n\n    print(\"=\" * 70)\n    print(\"✅ Demo completed!\")\n    print(\"=\" * 70)\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\n---\n\n## Key Takeaways\n\n1.  **Synergy of RL and Exit Rules**: An RL agent trains better when it doesn't have to worry about catastrophic losses (which are handled by `max_drawdown_percent`).\n2.  **Conservatism vs. Aggressiveness**: In this test, conservative rules limited the agent too much (only 17 trades), while aggressive rules allowed the PPO agent to realize its potential.\n3.  **Drawdown**: Using exit rules significantly reduces the maximum drawdown compared to a \"pure\" RL agent.\n\n---\n\n_Article prepared for the RLXBT community. More examples in the project repository._\n","coverImage":null,"status":"published","publishedAt":"2025-12-21T18:36:27.871Z","backtestResults":null,"researchManifest":null,"viewCount":334,"likeCount":0,"metaTitle":null,"metaDescription":null,"createdAt":"2025-12-21T18:36:27.875Z","updatedAt":"2026-09-09T13:14:39.285Z","author":{"id":"e1bbfa18-4766-4421-b638-f9c14a6b81b6","name":"RLXBT","picture":"https://lh3.googleusercontent.com/a/ACg8ocLU-AWuWIPI_04VXCHB_AfGIJAer4OwglsRBxupVzdauWYH3w=s96-c"},"tags":[],"comments":[],"isLiked":false,"isAuthor":false}