Closed-Loop Perception-to-Action Agent (clashBot)

Overview

An end-to-end vision-to-action agent for a partially observable real-time strategy game, operating under a closed loop of Context-Action-Feedback (CAF) while interacting with an emulator of a real-time strategy card game. The game is Clash Royale: a read-only perception pipeline turns gameplay into structured game state, a knowledge base built from the game wiki and 197 professional games supplies context, and a two-speed loop retrieves similar pro states, ranks candidate plays and hands the choice to a separate executor.

What I did

• Building an end-to-end vision-to-action agent for a partially observable real-time strategy game, operating under a closed loop of Context-Action-Feedback (CAF) while interacting with an emulator of a real-time strategy card game.
• Sourcing and labeling several custom datasets to train a YOLO model for localization and classification of assets used by the interface.
• Designing a two-tier control architecture that keeps the real-time loop on a fast reactive policy while retrieval-backed reasoning runs at a coarse cadence to combat retrieval latency that comes with each decision made.
• Designing a custom Embedding model for efficient retrieval utilizing a MOE-based framework to improve the base accuracy of the MMEB-V2 model while attempting to reduce the TFLOPs.
• Benchmarking a hand-authored heuristic policy against a learned policy trained offline on replay data, holding the perception stack fixed so the comparison isolates the decision layer.
• Built a read-only perception pipeline that turns match footage into game state: HUD readers for elixir, hand and clock, OCR of in-game deployment labels, YOLOv8 unit detection, and per-unit Kalman tracking on an 18×32 tile grid.
• Processed 197 professional games from 33 creator videos into 10,923 annotated game-state samples, then repaired, flagged and split them by video so near-duplicate states cannot leak across splits.
• Built hybrid retrieval on SQLite (structured-state kNN and BM25 fused with Reciprocal Rank Fusion) plus a 590-node knowledge graph; the fast decision loop runs at 41 ms p50 on CPU before any model is added.

Architecture

• Capture: recorded match video today; a live macOS window source through mss comes later. Perception is read-only and cannot send input.
• Screen: black-bar stripping, game-panel detection and per-frame readiness gating (match, menu, unreadable).
• HUD readers: elixir bar and digits, hand and next card by template matching against wiki card art, match clock with RapidOCR fallback, and tower HP; every read carries a confidence.
• Units: pretrained YOLOv8 detectors (two 85-class KataCR models with an ally/enemy channel) plus OCR of deployment labels, mapped onto an 18×32 tile grid by homography and tracked with per-unit Kalman filters.
• Events and state: own plays from hand change plus elixir drain, opponent plays from new tracks, an elixir simulator with measured drift, and an opponent deck and cycle tracker; GameState and PlayEvent records go to JSONL.
• Knowledge base: Markdown scraped through the Fandom wiki's MediaWiki API (126 cards, 42 evolutions, 16 heroes), 68 decks in 6 archetypes, 197 match files and 10,923 game-state samples.
• Clean data and knowledge graph: repair-and-flag cleaning, video-level splits (142/27/28 games), and a 590-node, 12,030-edge graph with COUNTERS, SYNERGY, IN_DECK and mined ANSWERED_BY relations.
• Retrieval: SQLite with FTS5/BM25, 644-dimensional structured state vectors with an HNSW index, fused by Reciprocal Rank Fusion; 3,654 pro-advice documents for the planner.
• Two-speed loop: fast path (decision trigger, kNN and knowledge-graph candidates, legality filter, ranker, executor) and a slow-path planner with six read-only tools (rule-based now; local Qwen LLM planned).
• Decision model: training data for LiquidAI d1-omni-600M exported in its Decision Index format (3,764 / 771 / 780 train/val/test); fine-tuning pending.

Stack

• Python
• YOLO
• PyTorch
• OOpenCV
• NNumPy
• RRapidOCR
• KKalman filtering
• SSQLite
• HHNSW (hnswlib)
• KKnowledge graphs
• RRAG
• mmodel evaluation and benchmarking
• BBeautiful Soup

Timeline

1. Oct 29, 2026

   Decision-model training data

   Exported training data for the LiquidAI d1-omni-600M decision model in its Decision Index format: 3,764 / 771 / 780 four-way choices (train/val/test). The request builder is shared with the runtime ranker so training and serving send identical inputs.

2. Oct 28, 2026

   Knowledge graph, hybrid retrieval and fast loop

   Cleaned the 10,923 game states with repair-and-flag rules and video-level splits, built a 590-node, 12,030-edge knowledge graph and 3,654 pro-advice documents, and seeded SQLite with BM25 and structured-vector kNN fused by RRF. The fast decision loop replays recorded games at about 41 ms p50 on CPU. Best simple baseline on the four-way choice: placement prior 0.37 / 0.36 (val/test) vs 0.25 random.

3. Sep 5, 2026

   Game-state dataset complete

   Extracted 10,923 game-state samples from 197 professional games and annotated every one with a situation read, reaction, confidence and tags; the validator reports 0 errors.

4. Sep 2, 2026

   Perception pipeline on pro footage

   Built the read-only perception package (HUD readers, deployment-label OCR, YOLOv8 unit detection, Kalman tracking, deck tracker) and ran it over 33 creator videos, producing 197 match files and 68 deck files.

5. Sep 2, 2026

   Knowledge base complete

   Scraped 126 cards, 42 evolutions and 16 heroes from the Clash Royale Fandom wiki through its MediaWiki API, with every QA check passing, and classified 19 popular decks into six archetypes.

Links