Closed-Loop Perception-to-Action Agent

Overview

An end-to-end vision-to-action agent for a partially observable real-time strategy game, operating under a closed loop of Context-Action-Feedback (CAF) while interacting with an emulator of a real-time strategy card game.

What I did

• Building an end-to-end vision-to-action agent for a partially observable real-time strategy game, operating under a closed loop of Context-Action-Feedback (CAF) while interacting with an emulator of a real-time strategy card game.
• Sourcing and labeling several custom datasets to train a YOLO model for localization and classification of assets used by the interface.
• Designing a two-tier control architecture that keeps the real-time loop on a fast reactive policy while retrieval-backed reasoning runs at a coarse cadence to combat retrieval latency that comes with each decision made.
• Designing a custom Embedding model for efficient retrieval utilizing a MOE-based framework to improve the base accuracy of the MMEB-V2 model while attempting to reduce the TFLOPs.
• Benchmarking a hand-authored heuristic policy against a learned policy trained offline on replay data, holding the perception stack fixed so the comparison isolates the decision layer.

Stack

• YOLO