PerceptIn progress

I built a camera app that draws what it is looking at.

Not the picture. The tracking. Corners it has locked onto, lines showing where they moved, boxes around things that are moving on their own.

Where it came from

There is a look you see in TouchDesigner work. A video with a web of thin lines drawn over it, points connected to points, boxes snapping onto whatever moves.

It is usually done after the fact, on footage, on a desktop.

I wanted it live, on a phone, on whatever the camera is pointed at right now.

What it is actually doing

Three things at once.

Shi-Tomasi picks the corners. Not every pixel is worth tracking, and a flat wall gives you nothing to hold onto. Corners are the points a tracker can find again in the next frame.

Lucas-Kanade follows them. Given a corner in this frame, it works out where the same corner went in the next one. That gives you the lines.

MOG2 handles motion. It builds a model of what the background looks like and flags whatever does not fit, which is how the boxes find things that are moving when the camera is not.

More recently ML Kit sits on top of that, doing on-device object detection, so the boxes know roughly what they are around rather than only that something changed.

The part that is hard

None of these algorithms are new. Running all of them on a live camera feed, on a phone, without the preview stuttering, is the entire problem.

Every frame has a budget. Miss it and the overlay lags behind what you are pointing at, which looks broken in a way that a still frame never shows you.

That is why the repository has a separate webcam validator. Getting the computer vision right on a desktop, on real pixels, is a different problem from getting it fast enough on a phone, and mixing the two makes both harder to debug.

tl;dr

Point the camera at something. Watch the machine work out what is worth looking at.