¶ Controlnet
There are 4 ways to incorporate an image into your AI art using Controlnet. In this example, I'm going to combine two images from Unsplash to create an artwork of a woman balancing on the shore at dawn.
¶ Initial Image
An Initial Image can be used with Controlnet. Unlike the Controlnet image preprocessors discussed below, Initial Images retain their colors.
* Please note, only the AI models based on SD 1.5 can use Controlnet with an Initial Image. SDXL models will be grayed out if you have an Initial Image selected.
I'm using the seashore background as an Initial Image in order to retain its color and lighting.
¶ Canny
Canny creates an edge map of your imported image. It works well for people and portraits as well as for landscapes and abstracts. In the image above, you can see how the Canny image processor detects the edges of the photo of balancing girl.
Below, the seashore photo has been processed with Canny.
As you work with Controlnet, you'll get a feel for which image processor works best for your project. In this case, I'll use the seashore image in Canny to reinforce the lines of the water and clouds.
For a demo that only uses Canny, click HERE.
¶ Depth
Depth creates a depth map of an image. Depth is great for portraits, but sometimes Canny works better, and sometimes the best bet is using both Canny and Depth together.
For a demo on portraits using Depth and/or Canny, click HERE
In this case, I'll use Depth and Openpose together to capture the pose of the woman.
¶ Openpose
Openpose creates a pose map to capture the pose of the person in your image. It works with full body poses as well as partial body poses. It even captures the position of the face and fingers.
¶ Weight
The weights for Controlnet preprocessors range from 0 to 2, though best results are usually achieved at 0.75. For this project, I'll use 0.50 because I have two inputs for each image.
- If your Controlnet images are not showing up enough in your rendered artwork, increase the weight.
- If your Controlnet images are overpowering your final artwork, decrease the weight.
¶ Prompt
Prompts and negative prompts are still important when using Controlnet. For this artwork, my prompt was:
beautiful woman wearing a bikini, beautiful face, dancing on the shore, as ocean waves crash behind her, best quality, beautiful, film grain
¶ AI Model
There are lots of AI models to choose from when using Controlnet. You can reimagine a photo in an anime style, or vice versa. You can see how your concept looks with different models. Since I was creating a photorealistic image, I'll use ICBINP SECO for this.
* Please note, if you are using an Initial Image, the SDXL models will be unavailable.
* If you switch from one model to another, the model may need to load, causing a delay.
Here is the finished artwork.
