Framework Inference
For ControlNet, the inference script generates images from text prompts defined in the config file. Note that, an image conditioning is required besides text prompt.
To enable the inference stage with ControlNet, configure the configuration files:
In the
defaults
section ofconf/config.yaml
, update thefw_inference
field to point to the desired DreamBooth inference configuration file. For example, if you want to use thecontrolnet/controlnet_infer.yaml
configuration, change thefw_inference
field tocontrolnet/controlnet_infer
.defaults: - fw_inference: controlnet/controlnet_infer ...
In the
stages
field ofconf/config.yaml
, make sure thefw_inference
stage is included. For example,stages: - fw_inference ...
Configure
prompts
andnum_images_per_prompt
fields ofconf/fw_inference/controlnet/controlnet_infer.yaml
. Setmodel.restore_from_path
to the ckpt generated from dreambooth training.
Remarks:
We have implemented an example processing function to extract segmentation map from a target image, and use that as the
conditioning to generate images. To utilize that function, please specify infer.control_image_preprocess=seg2img
. In other use cases, the input image specified at infer.control
should be the conditioning image
instead of target image, and set infer.control_image_preprocess=null
.