VITALS – Visual Intelligent Tactical Assistant for Low-Power Operations

Overview

Design, develop, and integrate a search and rescue (SAR) system that utilizes autonomous drones and an AI-powered ground control station running on NVIDIA Jetson hardware. The system uses multi-agent intelligence for mission planning, perception, and autonomous control in field-deployable operations.

What I did

• Owned the mapping and planning agent in a 4-agent SAR architecture (vision, retrieval, reasoning, mapping/planning), coordinating over MCP and ACP message contracts, on a 6-engineer team under Lockheed Martin sponsorship.
• Inherited and re-architected the terrain preprocessing pipeline: PostGIS-backed OpenStreetMap queries through SQLAlchemy, EPSG:3857 to EPSG:4326 reprojection, R-tree spatial indexing over a tiled search grid, with per-tile feature classification driving search priority.
• Redesigned multi-drone task allocation from a single greedy heuristic to a two-layer planner: A* global routing over a live costmap plus a local avoidance layer for obstacles absent from the OSM basemap, with non-overlapping timestamped sector assignment for persistent coverage.
• Specified the retrieval subsystem: bge-small-en-v1.5 encoder producing 384-dimensional embeddings into ChromaDB with cosine ranking and metadata filters, and ParquetDB cold archive for events aged out of hot storage.
• Targeted deployment on Jetson AGX Orin 64GB inside a 15–60 W configurable power envelope for disconnected field operation for the base station, and a Jetson Nano configured onto the drone for the vision edge.

Architecture

• Agent A — Vision: YOLOv11 and LLaVA, publishing scene facts over MCP.
• Agent B — Retrieval: ChromaDB with RAG.
• Agent C — Coordinator: LLM runtime on LLaMA 3, emitting ACP intents.
• Agent D — Mapping and Planning: C++/Python planners for maps, costmaps, A*/D*/coverage, and deconfliction.
• MAVLink Router (mavlink-routerd) and a Flask/WebSocket GUI adapter bridging Mission Planner.
• UAV fleet on ArduPilot with Pixhawk 2.4.8 flight controllers.
• Hardware: NVIDIA Jetson AGX Orin (275 TOPS INT8, 64GB LPDDR5, JetPack 6.x, CUDA 12.x, TensorRT 10.x), dual 4S LiPo, RFD900x radios, GPS M8N, and a RealSense depth camera.

Team

Team of 6.

• Justin Gamboa
• Brendan Rodriguez
• Alex Gershfeld
• Giorgio Torregrosa
• Stephen An
• David (Tyler) Jackson

Stack

• Python
• C++
• LangGraph
• YOLO
• NNVIDIA Jetson Orin AGX
• JJetson Nano
• CChromaDB
• PostgreSQL
• PPostGIS
• SSQLAlchemy
• MMAVLink
• AArduPilot
• Docker
• OOllama
• LLLaMA 3
• CCustomTkinter

Timeline

1. May 20, 2026

   MVP integration milestone

   Demonstrated MAVLink-based communication between Jetson ground station and drone.; Generated autonomous mission waypoints through multi-agent collaboration.; Delivered a fully integrated GUI interface linking AI reasoning, terrain mapping, and flight control.

2. Mar 5, 2026

   Path planning and terrain pipeline

   Develop spatial data ingestion pipeline using OpenStreetMap and PostGIS for terrain awareness.; Implement efficient path planning algorithms that respect drone constraints, terrain, and real-time telemetry.; Design coverage search patterns for multi-UAV coordination and obstacle avoidance.; Integrate geospatial visualization into GUI for operator situational awareness.

3. Jan 12, 2026

   Mission architecture and objectives

   Operator selects mission type (simulation vs live); Connects to MAVLink drone swarm; Defines Ground Control Station (GCS) location; Draws search polygon and generates terrain grid; Agent D (Mapping & Planning) generates optimized waypoints; Autonomous mission execution with live GUI feedback

Gallery

Links