Custom Avatar: Arbitrary Asset#

The goal of this process is to integrate your own custom character into the ACE pipeline. This documentation is for a humanoid avatar, but depending on your proficiency in Unreal Engine and 3rd party 3D tool(s), you can view this guide more as a hint, rather than strict requirements. Ultimately the ACE pipeline drives an avatar based on the 52 ARKit shapes. Everything else is only limited by your creativity.

Asset requirements#

For your own character to work with the pipeline you should have:

  • Shaded, textured and rigged 3D avatar with a face rig (simplest are 52 ARKit face blendshapes)

    ../_images/unreal-example-rigged-mesh.png

  • An idle animation

    ../_images/unreal-example-idle-animation.gif

  • An ARKit face animation:

    This animation will allow us to translate the Audio2Face-3D inputs to your character’s face rig in a later step.

    ../_images/unreal-example-animation-sequence.gif

    The animation can be created in Unreal or any 3D software that exports files compatible with Unreal (e.g. FBX). It must be 52 frames long. Each frame of the animation should correlate to the following ARKit poses. The order is important!

    1. eyeBlinkLeft

    2. eyeLookDownLeft

    3. eyeLookInLeft

    4. eyeLookOutLeft

    5. eyeLookUpLeft

    6. eyeSquintLeft

    7. eyeWideLeft

    8. eyeBlinkRight

    9. eyeLookDownRight

    10. eyeLookInRight

    11. eyeLookOutRight

    12. eyeLookUpRight

    13. eyeSquintRight

    14. eyeWideRight

    15. jawForward

    16. jawLeft

    17. jawRight

    18. jawOpen

    19. mouthClose

    20. mouthFunnel

    21. mouthPucker

    22. mouthLeft

    23. mouthRight

    24. mouthSmileLeft

    25. mouthSmileRight

    26. mouthFrownLeft

    27. mouthFrownRight

    28. mouthDimpleLeft

    29. mouthDimpleRight

    30. mouthStretchLeft

    31. mouthStretchRight

    32. mouthRollLower

    33. mouthRollUpper

    34. mouthShrugLower

    35. mouthShrugUpper

    36. mouthPressLeft

    37. mouthPressRight

    38. mouthLowerDownLeft

    39. mouthLowerDownRight

    40. mouthUpperUpLeft

    41. mouthUpperUpRight

    42. browDownLeft

    43. browDownRight

    44. browInnerUp

    45. browOuterUpLeft

    46. browOuterUpRight

    47. cheekPuff

    48. cheekSquintLeft

    49. cheekSquintRight

    50. noseSneerLeft

    51. noseSneerRight

    52. tongueOut

Setup in Unreal#

  1. Make sure you followed the Unreal Prerequisites guide closely.

  2. Create a new Unreal project. A blank Blueprint project is the recommended starting point.

    ../_images/unreal-new-project.png

    Note

    Windows has issues with paths longer than 256 characters. To keep the paths short, save your project in a location like: C:\UnrealProjects\MyProject.

  3. For the project to run, you’ll first need to install 2 plugins. Make sure to close the Unreal Editor before doing so. Use the NGC CLI to download these resources:

    ngc registry resource download-version "nvidia/ace/ace-unreal-renderer-http-api-plugin:0.1.0"

    ngc registry resource download-version "nvidia/ace/ace-unreal-renderer-stream-helper-plugin:0.1.0"

    Then place these plugins in a folder named “Marketplace” under the Plugins-folder of the project (e.g. C:\<PATH_TO_YOUR_UNREAL_PROJECT>\Plugins\Marketplace). If the “Marketplace” and “Plugins” folders don’t exist, create them.

    Note

    Beware of nested folders when unzipping:

    ../_images/plugin_unzip_issue_project.png

    The plugins are written and built for Unreal Engine 5.3. If you are using a different version, you will need to build the plugins yourself. There is also a chance, that some interfaces changed, so you might need to adapt the code to the new interfaces.

    # Windows example for compiling the plugin for Unreal 5.5
"C:\Program Files\Epic Games\UE_5.5\Engine\Build\BatchFiles\RunUAT.bat" -NoP4 BuildPlugin -Plugin="C:\UnrealPlugins\src\ace-unreal-renderer-stream-helper-plugin_v0.1.0\ACEUnrealRendererStreamHelper.uplugin" -Package="C:\UnrealPlugins\dist\ace-unreal-renderer-stream-helper-plugin_v0.1.0_UE5.5" -TargetPlatforms=Win64

# Linux example for compiling the plugin for Unreal 5.5
/home/ue4/UnrealEngine/Engine/Build/BatchFiles/RunUAT.sh -NoP4 BuildPlugin -Plugin="/tmp/UnrealPlugins/src/ace-unreal-renderer-stream-helper-plugin_v0.1.0/ACEUnrealRendererStreamHelper.uplugin" -Package="/tmp/UnrealPlugins/dist/ace-unreal-renderer-stream-helper-plugin_v0.1.0_UE5.5" -TargetPlatforms=Linux

  4. Open your project.

  5. Import your rigged character from an FBX or GLB file. You should get a “Skeleton”, at least one “Skeletal Mesh” and possibly other physics, material and texture assets.

    ../_images/unreal-example-imported-assets.png

  6. Create a Blueprint Actor for your avatar. In our case “BP_Robot” seemed fitting.

    ../_images/unreal-example-blueprint-creation.png

  7. Open the new Blueprint and drag in your avatar’s Skeletal Mesh.

    ../_images/unreal-example-avatar-bp.png

  8. In the same Blueprint, add an ACEAudioCurveSource to your avatar. This is where the audio from the ACE pipeline will be heard from.

    ../_images/unreal-example-add-audio-source.gif

  9. In the Event Graph of this Blueprint, drag out from the “Event BeginPlay” pip and create a “Set Actor” node from the ACE plugin. If you don’t see this option, double check that all plugins are installed correctly.

    ../_images/unreal-example-set-actor.png

  10. Drag out from the “Actor” pip of the “Set Actor” node and create a reference to “Self”. This small node network allows the ACE pipeline to find your avatar when the game starts.

    ../_images/unreal-example-begin-play.png

  11. Create a new Empty Level.

    ../_images/unreal-new-level.png

  12. Add your avatar blueprint to it and set up the environment to your liking.

    ../_images/unreal-example-level.png

  13. Set your new level as the default Level in the project settings. You can also assign custom Blueprints for the default Game Mode and Pawn in order to not use the standard first person character as an interface. We recommend a static camera instead.

    ../_images/unreal-example-defaults.png

  14. Import your idle animation into the project. Uncheck “Import Mesh” and make sure to specify your avatar’s skeleton asset as the “Skeleton”.

    ../_images/unreal-example-idle-import.png

  15. Setting up Face Retargeting using a Pose Asset.

    • Import your ARKit face animation the same as you just imported the idle animation.

    • Based on this ARKit face animation, you can create a Pose Asset.

      ../_images/unreal-create-pose-asset.png

    • Open the generated Pose Asset. It should have 52 values, each corresponding to a frame of the animation. Name these values according to the list of face poses mentioned in the requirements. You can do this quickly by copying the list of names from this page and pasting them to the list of poses in Unreal:

      ../_images/copy_paste_arkit_shape_names.gif

    • In the Asset Details of this Pose Asset, activate the “Additive” mode and make sure you are using the “Reference Pose”. Save the Pose Asset and close it.

      ../_images/unreal-example-additive-animation.png

      Note

      If you do not switch to Additive mode, you will run into a double transform issue later on.

    • Create a new Animation Blueprint using your avatar’s skeleton and open it.

      ../_images/unreal-example-animation-blueprint.png

    • In the AnimGraph tab, create an “Apply ACE Face Animations” node and get a reference to the Pose Asset we created earlier. Connect the output of the Apply ACE Face Animations into the Pose Asset. Get a reference to the imported idle animation and create an “Apply Additive” node. Using your idle animation as the base and the output of the pose asset as the additive, combines the ACE face animations with your idle animation. Connect the output of the Apply Additive node into the Output Pose.

      ../_images/unreal-example-anim-graph.gif

    • Hit compile and save the Animation Blueprint.

      ../_images/unreal-example-anim-graph.png

    • Open the blueprint of your avatar and assign the Animation Blueprint you just created to your Skeletal Mesh.

      ../_images/unreal-assign-abp.png

Now your character’s face can be controlled by the ACE animation pipeline.

../_images/unreal-example-result.gif
On this page