Launch: Synthetic Image Generation with DALL-E and GPT-4 Vision
Gathering images for use in training vision models can be time consuming, particularly if the objects you want to identify are rare and thus hard to capture. One solution to this problem is generating synthetic data, where you create artificial data based on real data.
You can then label the synthetic images and use them to help your model understand how to identify objects of interest. See our guide on creating synthetic by randomizing object location and background images or how to create synthetic data with Stable Diffusion for more ideas!
We are excited to announce a new Roboflow API endpoint that lets you add custom generated images directly to your Roboflow project, powered by DALL-E. In this guide, we are going to talk about how to generate synthetic images with the Roboflow synthetic image generation API.
Without further ado, let’s get started!
When Should You Use Synthetic Images?
Synthetic data can be used for a variety of purposes in computer vision. For use in dataset train splits, synthetic images are often used early in the life cycle of the model. They can be an easy and cost effective way to enlarge your dataset or fill gaps where you lack data.
Before using synthetic images we recommend searching for relevant images in Roboflow Universe, our collection of open source computer vision datasets. It is often more effective to start from similar images in real world datasets. When similar images are not available, synthetic images are a great alternative!
Introducing the Roboflow Synthetic Data Generation API
You can enhance your Roboflow dataset by piping generated images to your Roboflow projects. The Roboflow synthetic data generation API allows you to create synthetic data with OpenAI’s generative image models, such as DALL-E 3. A primary feature of our API is the ability to pass a base image to use as a description for the image prompt, which we generate using the vision capabilities of GPT-4.
The Roboflow synthetic data generation API provides one endpoint
POST https://api.roboflow.com/synthetic-imageThis endpoint accepts an image and a prompt and returns a synthetic image.
Let’s walk through an example of how to generate synthetic images with the API.
Setup OpenAI Key
To use this feature you will need to add your OpenAI API key to your Roboflow workspace settings.
Go to your Roboflow settings page. Then, select your workspace on the left.
Click Third Party keys and enter your OpenAI API key.
Finally, scroll down to the bottom of the page and click Save Settings.
Retrieve Configuration Values
We need to retrieve a few pieces of information to make a request to the synthetic data generation API.
In your workspace settings, copy your Roboflow API key from the Roboflow API section. Follow our authentication documentation if you need help finding it. In the examples below, replace ROBOFLOW_API_KEY with your key.
In order to save the generated image to your project, you will need to send the project URL with the request.
To retrieve your project URL, go to the Version page associated with your trained model in the Roboflow dashboard. At the top of the page, the string before the /VERSION_NUMBER is the project URL (also called project ID).
Follow our guide on how to retrieve a project URL for more information.
Generate an Image from a Prompt
You can generate a new image for your Roboflow project by simply passing a prompt.
curl -X POST "https://api.roboflow.com/synthetic-image?api_key=ROBOFLOW_API_KEY" \
-H 'Content-Type: application/json' \
--data \
'{
"project_url": "YOUR_PROJECT_URL",
"prompt": "An image of a raccoon in a tree."
}'Example Response (from DALL-E 3 model):
{
"created": 1701126433,
"data": [
{
"revised_prompt": "Generate a photorealistic image of a raccoon perched on a tree branch. It should have the viewpoint of a Sigma 85mm f/8 lens, subtly interspersed with minute visual aberration and a minimal layer of camera noise throughout the entire image.",
"url": "https://oaidalleapiprodscus.blob.core.windows.net/private/org-.../user-.../img-....png?..."
}
]
}You can find the resulting image in the Unassigned or Annotate section of your Roboflow project.
Generate from a Roboflow Image ID
A more detailed way to generate an image is from an existing one. This method can be combined with a prompt to create a specific variation of that image.
To retrieve an image ID from Roboflow, you can use the Roboflow Image Search API, or open a specific image in Roboflow.
To open a specific image, go to the images tab in your Roboflow project and select an image.
From here, you can copy the image ID out of the browser URL, or go to the Raw Data tab and find the id field.
Once you have the image ID, you can make the request.
curl -X POST "https://api.roboflow.com/synthetic-image?api_key=ROBOFLOW_API_KEY" \
-H 'Content-Type: application/json' \
--data \
'{
"project_url": "YOUR_PROJECT_URL",
"prompt": "At nighttime.",
"image": "IMAGE_ID",
"image_type": "id"
}'Example Response (from DALL-E 3 model):
{
"created": 1701127521,
"data": [
{
"revised_prompt": "A photo-realistic image showing two industrial workers, [...] on a metallic staircase within an atmospheric industrial setting at night time, they appear engaged [...]",
"url": "https://oaidalleapiprodscus.blob.core.windows.net/private/org-.../user-.../img-....png?..."
}
]
}
You can find the resulting image in the Unassigned or Annotate section of your Roboflow project.
Managing Synthetic Data
Later in a computer vision project’s timeline, you will want to phase out the synthetic images in place of real images taken from your cameras running in production. With the Roboflow Inference package, you can start collecting production images even before you have a model trained! Another way to easily gather images from production is with Roboflow Collect.
After you have enough images from production and you want to remove the synthetic images, you can filter your images by tag in the Images section in Roboflow.
All images generated with our API will have a tag starting with synthetic-. Then you can Select All, and Remove from Project.
Conclusion
The tools available for synthetic image generation are improving every day! We encourage you to test out generated images with your Roboflow vision models. For more ways to use synthetic images with Roboflow see the following guides: