Machine vision applications with wearable devices

Using machine vision in industry

XR Tech is a UK based solutions provider for extended reality (XR) technologies, digital transformation, and automation in industry. XR Tech uses devices such as RealWear, Hololens, Oculus, and others to assist and augment situational awareness, machine vision and industrial operational activities.

XR Tech equips workers with a body mounted camera, and streams the rich media data to machine vision models they trained on the Roboflow platform to produce a “smart” safety system. This machine vision camera system performs a range of operations, from detecting personal protective equipment (PPE) – ensuring all workers stay within safety code; and to identify high risk hazards such as loose rubble fall locations and heat exhausts.

All of these machine vision examples are performed within XR Tech’s mobile application and can be deployed to various locations. High risk situations, such as workers not wearing proper PPE, can be responded to instantly and with higher precision, due to the fast performance of the machine vision pipeline. This smart safety system protects workers, and creates a proactively managed ecosystem for high risk industrial environments.

Building the machine vision system

Before XR Tech could train highly effective models, the XR Tech team needed to understand the principles behind good training practices and data hygiene. The XR Tech onboarding process focused on processes such as how to create a strong machine vision labeling ontology, how to annotate data effectively, and understanding which augmentations would add to – instead of detract from – the training set.

Data Collection for machine vision applications

To speed up the development of the PPE object detector models, XR Tech reduced the need for manual annotation and data collection by leveraging an existing hard hat dataset.

This initial model could then be deployed into controlled testing environments to understand which scenes cause the machine vision algorithm to fail. For example, when large groups of people wearing boots stood too closely together, the model would think the space between boots was another boot.

Equipped with an understanding of the failure points, XR Tech focused data collection effort on targeted scenes to address breakpoints – instead of general data collection efforts. This led to faster model improvement as each new set of data points responded to a specific problem.

Annotating for machine vision algorithms

The initial models trained from existing data also allowed for faster annotating when applicable through the label assist tool. However, the classification labeling needed to start from scratch as training a classifier uses a different approach than object detection.

Preparing a machine vision dataset

XR Tech’s initial model was a grab bag of general PPE classes such as hardhats, vests, and goggles. Using the health check tool to see which individual classes needed improvement, they created tailored projects (organized dataset buckets within a Roboflow workspace) that would only contain data relevant to the class they were improving. For example, XR Tech created a class called “clear_goggles_only” that contained data in which individuals wore clear goggles but no other PPE gear.

It was critical that these individual datasets were free of other PPE gear as any equipment that the final model needs to be trained on must be annotated when present in an image. Individual projects would be merged into groupings such as “small_ppe_gear” to see how the model performed without major gear present before being merged into the final model used in deployment.

Machine vision model deployment

The XR Tech mobile app can record data from the camera live stream and periodically sends a base64 encoded image to their trained model hosted via Roboflow’s remote endpoint. Inference results get returned to the mobile app as JSON data, and controllers with the app signal appropriate parties based on what the machine vision model inferred.

From images to inferences for machine vision

XR Tech is a great example of machine vision having real world impact in industry. The mix of cutting edge technology and machine vision best practices has created new breakthroughs that were previously not possible. Discover an image dataset and build your application today.