Motion Detection with Computer Vision

With this guide, you'll learn how to use computer vision to only identify moving objects in a video. Here is a hosted demo to try out.

Object detection models can be used on a live video feed, pre-recorded video, youtube video, webcam, etc. Lets say you do not want to detect all objects in a video, you just want to detect moving objects.

Use Cases for Motion Detection

You may want to have a security camera and not pick up objects that are in the background or you may want to set up alerts with a pet camera that notifies you when your pet might be getting into trouble (or doing something cute).

In the above example, I am running an object detection model from Roboflow Universe and only want to run it on moving objects.

How it works

At a high level, here's how it's done:

1. You start the inferenceJS worker just like normal but separately run a new function captureMotion.
2. captureMotion will look back at the previous frame and the current frame and create a box that contains all the differences detected over a certain threshold.
3. Then the object detection will be filtered down to only what is inside that box
4. This repeats every 500 milliseconds

You start this by adding a setInterval function before the worker starts.

video.onplay = function() {
  
  setInterval(captureMotion, 500);

  inferEngine.startWorker(model_name, model_version, publishable_key, [{ scoreThreshold: confidence_threshold }])
    .then((id) => {
      // Start inference
      detectFrame();
  });
}

Global Variables

We will add a bunch of variables that can be tweaked. Generally, you will only want to mess with the pixelDiffThreshold and scoreThreshold.

pixelDiffThreshold can be thought of as the difference before you consider it a change has occurred at all.

scoreThreshold can be thought of as the number times a difference has happened.

// The canvas's are made and prepped for differencing
var isReadyToDiff = false;

// The pixel area of change
var diffWidth = 64;
var diffHeight = 48;

// Change these based on your preference of motion
var pixelDiffThreshold = 32;
var scoreThreshold = 16;

// The dict containing the box dimensions, width, and height to draw
var currentMotionBox;

// A canvas for just the difference calculation
var diffCanvas = document.createElement('canvas');
diffCanvas.width = diffWidth;
diffCanvas.height = diffHeight;
var diffContext = diffCanvas.getContext('2d', { willReadFrequently: true });

Let's take a quick break and look at a cat video example before continuing.

Motion Detection Logic

This is captureMotion without any of the extra drawing box code. We get an image representing the difference between frames, shrink down the image into a canvas with size diffWidth and diffHeight, and finally put that into an array for comparing.

function captureMotion() {

	diffContext.globalCompositeOperation = 'difference';
	diffContext.drawImage(video, 0, 0, diffWidth, diffHeight);
	var diffImageData = diffContext.getImageData(0, 0, diffWidth, diffHeight);

	if (isReadyToDiff) {
		var diff = processDiff(diffImageData);

		if (diff.motionBox) {
			// Draw rectangle
		}
		currentMotionBox = diff.motionBox;
	}

}

Then we process the difference with processDiff to determine if motion has occurred and set the motionBox if returns something.

If the score gets above your set scoreThreshold, a motion is detected and it is returned. The rest of code is just determining the box dimensions and returning it.

function processDiff(diffImageData) {
	var rgba = diffImageData.data;

	// Pixel adjustments are done by reference directly on diffImageData
	var score = 0;
	var motionPixels = [];
	var motionBox = undefined;
	for (var i = 0; i < rgba.length; i += 4) {
		var pixelDiff = rgba[i] * 0.3 + rgba[i + 1] * 0.6 + rgba[i + 2] * 0.1;
		var normalized = Math.min(255, pixelDiff * (255 / pixelDiffThreshold));
		rgba[i] = 0;
		rgba[i + 1] = normalized;
		rgba[i + 2] = 0;

		if (pixelDiff >= pixelDiffThreshold) {
			score++;
			coords = calculateCoordinates(i / 4);

      motionBox = calculateMotionBox(motionBox, coords.x, coords.y);

			motionPixels = calculateMotionPixels(motionPixels, coords.x, coords.y, pixelDiff);

		}
	}

	return {
		score: score,
		motionBox: score > scoreThreshold ? motionBox : undefined,
		motionPixels: motionPixels
	};
}

Draw the Motion Detection Container

All the code is there, but we still have to do the actual filtering. That is done on the drawing level.

In short, if the motion is detected and the prediction is inside the box created, show that prediction.

function drawBoundingBoxes(predictions, ctx) {

  if (currentMotionBox != undefined) {

    // Draw motionBox

    for (var i = 0; i < predictions.length; i++) {

      if (confidence < user_confidence || !isCornerInside(motion_x, motion_y, motion_width, motion_height, x, y, width, height)) {
        continue;
      }

Here is final example video with bunnies. They tend to freeze up and suddenly move so the motion detection can be shown.

Conclusion

It's a bit confusing but much easier to grasp if you play with the code and change things to see what they do.

I encourage you try out different models and change the sliders for "Pixel Difference Threshold" and "# of Changed Pixels Threshold" and see how they change the motion detection.

The Github repository includes instructions for how to run it on your server with Vercel, Heroku, and Replit.

Cite this Post

Use the following entry to cite this post in your research:

James Steinberg. (Sep 24, 2024). Motion Detection with Computer Vision. Roboflow Blog: https://blog.roboflow.com/motion-detection-computer-vision/

Discuss this Post

If you have any questions about this blog post, start a discussion on the Roboflow Forum.