I have an IoT-controlled gate in my office. I can open and close it from Apple Home through Home Assistant. It’s working fine, never had a problem with it… technically.

There were two instances, though, that I somehow left it open, or open button was accidentally pressed. I only found out the next day that I left the gate opened all night. Both times, I slowly walked into the office hoping nothing was missing.

Thankfully, the two times it happened nobody took anything.

My gate setup is not smart enough that it knows the actual state of the gate (open or close).

But I have a CCTV camera pointing at the gate.

Making it smart

I’ve been looking for practical use of small models like gemma-4-e4b (supports vision!) which easily fits the GPU in my homelab server. I have it exposed via llama.cpp with llama-swap, an OpenAI-compatible endpoint, and it’s a vision model accessible to any service running in my lab.

I’ve thought of pulling a snapshot every minute and have it process the image using a prompt, then automatically close the gate if left open for 15 minutes.

How it works

Detection is one vision-model call, once a minute.

Each cycle pulls a keyframe over RTSP, crops to the lower-center of the gate, and sends the crop to the model. Temperature 0, thinking off. About a second per call.

You are looking through a carport at a sliding driveway gate, at the lower-center where the driveway meets the exit.

If you see open road/pavement/ground leading outside (gate slid away) it is OPEN.

If a solid row of vertical metal slats/bars blocks the driveway at ground level it is CLOSED.

A gate only partly open (any visible gap to the outside) also counts as OPEN.

Thin bright light-lines BETWEEN the vertical bars are still CLOSED.

When unsure, answer OPEN.

Respond with STRICT JSON only, no prose:
{"state":"OPEN"|"CLOSED","confidence":0-100,"reason":"<short>"}

“When unsure, answer OPEN” is the fail-safe. Better to have a false-positive open, than a false positive close.

Even a partial gap counts as open.

The reply is JSON, so I log a state, a confidence, and a reason. One frame decides nothing. A change needs two polls in a row to agree.

Making it safe

Moving a physical gate is the part to be careful with. The alert always lands before anything moves.

• Home Assistant pages me after the gate has been open for 15 minutes.
• It keeps checking for the next 3 minutes. If the gate is still open for three consecutive checks, it triggers the gate to close.

There is currently no obstruction sensor. So before it closes, it takes a fresh frame and asks:

This is a driveway gate opening that is about to be auto-closed.

Is there ANY person, animal, or vehicle in or partly in the gate opening / directly in its path (anything that could be hit by a closing gate)? Be cautious.

Strict JSON: {"blocked":true|false,"what":"<short>"}

If something is blocking, it does not close.

I watched it run on its own once. After 18 minutes, the path was clear, so it closed the gate.

Making it super safe

I still have doubts about the prompt. Even if it double-checks or triple-checks it could still be wrong and cause an accident.

I already have an IR obstacle sensor that can be wired into the gate mechanism to stop it if something is in the way, I just need the time to install it.

Unfortunately, I can’t ask Claude/Codex to do it for me 😛

It’s on my to do list!