Launch: Collaborative Annotation Updates

Roboflow’s Collaborative Annotation and Label Only User features have helped over 100,000 users annotate more than 100 million images and today we are excited to share new updates that will help teams speed up their annotation and data management efforts even more.

The Collaborative Annotation workflow makes managing the pipeline of uploading, assigning, and reviewing easy for large teams working on multiple projects. With computer vision and machine learning teams growing the number of projects they manage, we worked with customers to understand more advanced features to help make the labeling workflow more efficient.

API Upload Batches and Assign Jobs

Teams building their computer pipeline with Roboflow can now accomplish more tasks via API. You’re able to assign Jobs, get information on job statuses, and see relevant data of all, or individual, Batches.

$ curl --location --request GET 'https://api.roboflow.com/${WORKSPACE}/${PROJECT}/batches?api_key=${ROBOFLOW_API_KEY}'

{
    "batches": [
        {
            "name": "Uploaded on 11/22/22 at 1:39 pm",
            "numJobs": 2,
            "images": 115,
            "uploaded": {
                "_seconds": 1669146024,
                "_nanoseconds": 818000000
            },
            "id": "<BATCH_ID 1>"
        },
        {
            "numJobs": 0,
            "images": 11,
            "uploaded": {
                "_seconds": 1669236873,
                "_nanoseconds": 47000000
            },
            "name": "Upload via API",
            "id": "<BATCH_ID 2>"
        }

You’ll be able to build automated workflows to manage your projects without having to do any manual effort. This feature is especially useful when using active learning to bring data back into Roboflow for continuous model improvement.

Once you’ve set thresholds for conditional upload logic and upload those images to your Roboflow workspace, you can trigger Job assignments instantly.

View Batch Metrics and Details

For users on paid plans with access to our Review features and Role Based Access Control, in the Board View you can see aggregate metrics for the entire project by Stage and Job. The new aggregate metrics for each column makes it easy to see progress across all stages of the labeling workflow.

By hovering over the top section of any given stage, you'll see the metrics section highlight and your cursor offer the ability to click for more detail information.

Once you click into the Stage, you will see a simplified table view of each Labeler, their number of jobs in the stage, how many images they have in total, and how many images they have labeled.

This detailed view gives you insight into progress made by your team and allows you to see bottlenecks in your process. Later in this post we will show you how to use this information to Reassign work to other Labelers or Reviewers.

View Workloads by Labeler and Reviewer

Paid plan users managing multiple different projects with tens or hundreds of labelers can be difficult to understand the speed of progress and where there are bottlenecks. Ensuring workloads are balanced can help speed up the process by reducing downtime and increasing utilization.

You can view the progress of any specific Labeler or Reviewer by toggling the filters at the top of the Annotate page.

Once you’ve selected a Labeler, Reviewer, or combination of the two, you’ll see the progress on each individual Job assignment and aggregate progress across stages in the Annotation workflow.

From here, you’re able to identify jobs that could use more resources and Reassign teammates to help out.

Search and Assign Reviewers or Labelers

For users with access to Review and Label roles with large teams, your list of potential teammates can be long so we’ve added a new search bar when assigning Labeler or Reviewer roles to Jobs.

This new feature is useful as you work to balance Jobs across team members. Once you click into the Assignment section of a job, the search bar is available to Reassign the Job to any Labeler or Reviewer.

This gives you the opportunity to triage and troubleshoot bottlenecks in real-time. You’ll be able to coordinate workloads and optimize Jobs for maximum efficiency.

Managing Distributed Labeling Projects with Roboflow

We are excited to give users more control over their data labeling operations and help make managing large scale dataset creation projects easier for distributed teams.

Let us know how these new features are helping you scale up labeling efforts and ensuring teams are efficient by posting your thoughts, questions, and suggestions in the forum.

Cite this Post

Use the following entry to cite this post in your research:

Trevor Lynn. (Dec 16, 2022). Launch: Collaborative Annotation Updates. Roboflow Blog: https://blog.roboflow.com/collaborative-annotation-updates/

Discuss this Post

If you have any questions about this blog post, start a discussion on the Roboflow Forum.