Framework Inference

For DreamBooth, the inference script generates images from text prompts defined in the config file, similar to section 5.7.3. Note that, dreambooth is a fine-tuning model based on diffusion models to link a special token with certain subject, so make sure the special token you trained on is included in the text prompt. For example, a photo of sks dog sleeping.

To enable the inference stage with DreamBooth, configure the configuration files:

  1. In the defaults section of conf/config.yaml, update the fw_inference field to point to the desired DreamBooth inference configuration file. For example, if you want to use the dreambooth/text2img.yaml configuration, change the fw_inference field to dreambooth/text2img.

    Copy
    Copied!
                

    defaults: - fw_inference: dreambooth/text2img ...

  2. In the stages field of conf/config.yaml, make sure the fw_inference stage is included. For example,

    Copy
    Copied!
                

    stages: - fw_inference ...

  3. Configure prompts and num_images_per_prompt fields of conf/fw_inference/dreambooth/text2img.yaml. Set model.restore_from_path to the ckpt generated from dreambooth training.

Remarks:

Please refer to DreamBooth Training , the inference stage of DreamBooth should be conducted subsequent to the DreamBooth conversion process. This conversion transforms the DreamBooth ckpt into a ‘.nemo’ format and meanwhile remapping the parameter keys into Stable Diffusion style, allowing for a consistent inference pipeline.

Previous Training with Predefined Configurations
Next Model Export to TensorRT-LLM
© Copyright 2023-2024, NVIDIA. Last updated on Feb 22, 2024.