Important

You are viewing the NeMo 2.0 documentation. This release introduces significant changes to the API and a new library, NeMo Run. We are currently porting all features from NeMo 1.0 to 2.0. For documentation on previous versions or features not yet available in 2.0, please refer to the NeMo 24.07 documentation.

Framework Inference#

For ControlNet, the inference script generates images from text prompts defined in the config file. Note that, an image conditioning is required besides text prompt.

To enable the inference stage with ControlNet, configure the configuration files:

In the defaults section of conf/config.yaml, update the fw_inference field to point to the desired DreamBooth inference configuration file. For example, if you want to use the controlnet/controlnet_infer.yaml configuration, change the fw_inference field to controlnet/controlnet_infer.
```
defaults:
  - fw_inference: controlnet/controlnet_infer
  ...
```
In the stages field of conf/config.yaml, make sure the fw_inference stage is included. For example,
```
stages:
  - fw_inference
  ...
```
Configure prompts and num_images_per_prompt fields of conf/fw_inference/controlnet/controlnet_infer.yaml. Set model.restore_from_path to the ckpt generated from dreambooth training.

Remarks:

We have implemented an example processing function to extract segmentation map from a target image, and use that as the conditioning to generate images. To utilize that function, please specify infer.control_image_preprocess=seg2img. In other use cases, the input image specified at infer.control should be the conditioning image instead of target image, and set infer.control_image_preprocess=null.