01 The Problem
A motion sensor can tell me something moved outside the house. It cannot tell me whether I should care. A delivery driver, a family member coming home, a fox, a branch in the wind, and a person in a mask at 2 a.m. all trip the exact same binary sensor. The traditional fix is more sensors and more rules.
I wanted to try something different: drop a multimodal LLM in as a reasoning layer between the raw event stream and the smart-home logic, and let it make the judgment a human would make glancing at the camera feed. It is, very honestly, a proof of concept — the v4 in the automation's name and a handful of toggled-off steps are the tell that I'm still tuning it.
02 AI as a Reasoning Layer, Not a Detector
Most "AI camera" features answer one question — is there a person? — and stop. That's object detection, and it fires identically for the mail carrier at noon and the stranger at 3 a.m. The bet here is that the valuable question is the contextual one a person answers without thinking: given who's home, what time it is, and what I'm looking at, should I be worried?
┌────────────────────────────────────────┐
│ RAW EVENT STREAM "something moved" │
│ motion sensors · driveway count │
└─────────────────┬──────────────────────┘
▼
┌────────────────────────────────────────┐
│ REASONING LAYER (LLM) "should I care?"│
│ vision frames + live home state │
│ + guard-role heuristics → judgment │
└─────────────────┬──────────────────────┘
▼
┌────────────────────────────────────────┐
│ DETERMINISTIC LAYER "do the thing" │
│ helper entities → dashboards · alerts │
└────────────────────────────────────────┘
The model isn't asked to detect — it's asked to reason, with the same situational inputs a human guard would have. Keeping those three layers clean is the whole design: dumb sensors in, soft judgment in the middle, and a rule engine downstream that only ever sees safe, enumerated values.
03 The Pipeline
It's a single Home Assistant automation running top to bottom in single mode, so it won't trample itself if motion keeps tripping. Motion (or a driveway active-count change) fires it; a snapshot is taken; the frames go to the model; the structured verdict is written back into the home.
TRIGGER front motion / driveway count
│ (a backyard-motion trigger exists, disabled)
▼
PICK CAMERA (Jinja) front if front-motion, else backyard
│
▼
DELAY 1s ── let the scene settle
│
▼
camera.snapshot → timestamped frame on disk
│
▼
llmvision.image_analyzer
┌──────────────────────────────────────┐
│ images: ALL cameras, not just trigger │
│ prompt: "security guard" + live state │
│ response_format: json (schema-locked) │
└──────────────────────────────────────┘
│ → ai_result
▼
WRITE HELPERS
input_select.security_threat_level
input_select.security_action_needed
input_text.security_last_description (≤255)
│
▼
IF not is_false_positive → "real event" branch
├─ notify "🚨 CONFIRMED THREAT" (disabled)
└─ llmvision.create_event → timeline entry
A few details worth calling out:
- Multi-camera, holistic. The snapshot is of the triggering camera, but the analyzer is handed every camera's frame. Motion often trips one sensor while the real activity is in another camera's view — sending the whole scene lets the model see what a single feed would miss.
- A one-second settle delay. Snapshotting on the leading edge of motion tends to catch a blur or an empty frame; a short pause buys a cleaner, more interpretable image.
- The helpers are the contract. The real output isn't the notification — it's the helper entities. Everything else in the house (dashboards, downstream automations, alerting) reads those, so the AI step stays decoupled from whatever reacts to it.
- Actively iterated. The push notification and a couple of branches are switched off right now — deliberate, while I tune the prompt and don't want a half-cooked model deciding whether my phone buzzes at night.
04 Context Injection
The piece I'm proudest of is the prompt. It doesn't just ask is there a threat in this image. It injects live home state at the moment of the event and hands the model heuristics to reason with — so the same visual scene gets judged differently depending on the world around it.
INJECTED PER EVENT (via Jinja)
• local time
• time-of-day bucket late night / morning /
afternoon / evening / night (computed)
• alarm panel state armed / disarmed
│
▼ framed as a "security guard"
HEURISTICS HANDED TO THE MODEL
• private, isolated residence
• late at night, nobody about → raise suspicion
• alarm armed-away → ANY person raises threat
• door already unlocked → someone may be expected
• read attire: suit lowers · hoodie / covered
face raises
The effect: a person in the yard at 2 p.m. with the alarm off and a delivery uniform reads very differently from a person at 2 a.m. with the alarm armed-away and a covered face — even though, to a motion sensor, they're the identical event. The context is fed in fresh on every trigger rather than baked into static rules.
05 Structured Output & the Safety Bias
A reasoning layer is only useful to a smart home if its output is safe to act on. Free text from an LLM isn't — you can't branch an automation on a paragraph. So the model is pinned to a strict JSON schema (response_format: json, with additionalProperties: false so it can't invent fields).
{
"threat_level": low | medium | high,
"person_type": family | courier | unknown,
"action_needed": none | verify | alert,
"is_false_positive": boolean,
"description": string (< 100 words),
"short_description": string
}
most fields required · additionalProperties: false
The enums are the whole point. Because threat_level can only be low, medium, or high, the downstream input_select helpers have exactly those options, and automations can pattern-match on them with total confidence. The messy, probabilistic judgment happens inside the model; what comes out is a small set of known values a rule engine can treat as gospel.
One schema rule does more work than the rest: is_false_positive may be true only when the scene is clearly nothing — but never when threat_level is medium or high. That's a deliberate fail-safe. The expensive mistake in security isn't a false alarm, it's dismissing a real one — so the model is forbidden from simultaneously saying "this is a threat" and "...but ignore it." The bias is encoded in the instructions, not bolted on afterward.
06 Reflections & Honest Limitations
What excites me is how cleanly the three-layer split worked. Dropping an LLM in as the middle reasoning tier — fed live home state, constrained to enumerated output — turned a binary motion event into a context-aware judgment without me writing a single hand-tuned rule about hoodies or armed-away states. The model does the soft reasoning; the schema makes that reasoning safe to hand to deterministic automation. Context injection plus structured output is, I think, a genuinely reusable pattern for bolting AI judgment onto any rule-based system.
The limitations are real and I won't dress them up. It's tuned to one property — every heuristic in the prompt is specific to this house and wouldn't generalize without a rewrite. Reliability lives in prose: behavior, including the safety bias, depends entirely on prompt wording, and there's no test harness catching a regression when a model updates. And it calls a vision LLM on every motion event — a busy day of deliveries and wind is a lot of inference, in money and in the few seconds between motion and verdict.
The honest dealbreakers for anything beyond a personal experiment: it sends frames of the home's exterior to an LLM provider, and nothing learns — misjudge the same driver every afternoon and it'll keep doing it, short of editing the prompt. For what it set out to be — a test of whether a multimodal model can sit between dumb sensors and smart-home logic and make a useful, structured, situational call — it works, and the architecture is the part I'd keep.