SPOTlight

Overview

SPOTlight is an operator console built at Knight Hacks IX for NextEra Energy's SPOT challenge: help plant operators understand what the Boston Dynamics SPOT robot is seeing, live and after a mission, without becoming technical experts. It shows what actually went wrong, explains why in plain language, and says what to do next. Its rule is gates first, model second: explainable rules and a small classifier decide what deserves attention, a large model explains it, and a deterministic checker decides whether that explanation is shown.

What I did

• In NextEra's challenge kit, 26 of the 27 SPOT missions are marked COMPLETED, yet 21 of those 26 contain failed actions, and 14 of the 15 HIGH or CRITICAL anomalies happened inside "completed" missions; the console is built to show what actually went wrong, explain why in plain language, and say what to do next.
• System 1 triages every event with Laya (421M, ModernBERT-large), fine-tuned with PyTorch on ROCm on an AMD Radeon 8060S, next to a rules validator modelled on ISA-18.2 alarm rationalization; the best fine-tune reached a macro-F1 of 0.805 on held-out real missions and 0.846 on counterfactuals.
• System 2 explains escalated flags with Qwen3.8-27B (8-bit GGUF) in llama.cpp built with HIP for an AMD Radeon Pro W7900D, at about 22 tokens/s, and a deterministic checker requires every number and ID in a claim to appear in its cited rows.
• On 38 held-out flags from 8 unseen missions, 36 answers (94.7%) were accepted on the first attempt, the other 2 fell back to a labelled template, and every answer served passed the schema, citation, number and name checks; latency was 11.2 s at p50 and 15.0 s at p95.
• Partial JSON from the model streams as draft snapshots over SSE through the FastAPI backend and Next.js, so drafts labelled "unverified" appear in about 3 seconds; the checked answer then replaces them, and only checked answers are ever stored.
• Retrieval runs on Neon Postgres with pgvector: 503 documents embedded with Qwen3-Embedding-0.6B on the AMD box, hybrid vector and full-text search, and a 574-node knowledge graph stored as plain tables and walked with WITH RECURSIVE.
• A gauge photo reading counts only when two independent readers agree within 2% of the dial's span; otherwise the operator sees the photo and is asked for a retake or a human reading.

Architecture

• System 1, triage on every event: Laya (421M, ModernBERT-large) answers typed questions in one forward pass with calibrated probabilities; an ISA-18.2-style rules validator sets a priority floor that Laya can raise but never lower.
• System 2, explanations for escalated flags: Qwen3.8-27B (8-bit GGUF) in llama.cpp built with HIP, given one prepared context packet from a read-only MCP server and no tool calls, answering in Orbit's format (question, answer, cited evidence, recommended action, confidence).
• Deterministic checker: every number and ID in a claim must appear in its cited rows; a rejected answer is retried, then replaced by a template labelled as a template.
• Streaming: llama-server's partial JSON is relayed as SSE draft snapshots through FastAPI and Next.js; drafts are labelled "unverified" and only checked answers are stored.
• Retrieval: Neon Postgres with pgvector holds 503 documents embedded with Qwen3-Embedding-0.6B, searched with combined vector and full-text matching, plus a 574-node knowledge graph walked with WITH RECURSIVE.
• Vision: code readers for dial gauges, thermal colorbars, spectrograms and Orbit screenshots turn a photo into citable readings and an annotated image; no vision-language model computes a number.
• Deployment: Next.js frontend, FastAPI backend and MCP server on Vercel; one Neon database for operations, RAG documents, the knowledge graph and audit logs; the LLM, embedding model, Laya and System 2 under supervisord on the AMD box, reached through Cloudflare tunnels.
• Compose stack adds a LangGraph orchestrator and a LiteLLM gateway; SPOTbot chat answers through the Gemini API; every service has a Dockerfile under one docker-compose.yml, and CI runs 14 checks on every PR.

Team

Team of 4.

Stack

• Python
• PyTorch
• AAMD ROCm
• lllama.cpp
• FastAPI
• Next.js
• TypeScript
• React
• Tailwind CSS
• MModernBERT (Laya)
• QQwen
• HHugging Face
• PostgreSQL
• NNeon
• ppgvector
• RRAG
• MMCP
• LangGraph
• LLiteLLM
• Gemini API
• OOpenCV
• SServer-Sent Events
• Docker
• VVercel
• Claude Code

Timeline

1. Oct 11, 2026

   Streaming drafts and a live patrol

   Streamed Qwen's partial JSON into the UI as drafts labelled "unverified", always replaced by the checked result; only the checked result is stored. System 2's /health now runs a real one-token model probe after llama.cpp silently returned errors while /health still said OK. Live patrol defaults to a mock Mission 1 so the demo works without the AMD tunnel.

2. Oct 10, 2026

   Two-tier AI on AMD ROCm

   System 1 is Laya plus a rules validator on every event; System 2 is Qwen3.8-27B (Q8_0 GGUF) served by llama.cpp on a Radeon Pro W7900D, with a deterministic post-check. The held-out run measured p50 11.2 s and p95 15.0 s, missing the 15 s budget by 3 ms; the team kept 8-bit quality and fixed the experience with streaming instead.

3. Oct 9, 2026

   Reading the data before designing

   Loaded NextEra's challenge kit into pandas: 26 of 27 missions are marked COMPLETED, yet 21 of those 26 contain failed actions, and action success falls to 33% at or below 25% battery. Chose an operator dashboard for NextEra's SPOT challenge, with live replay and post-mission review.

Gallery

Links