Inkling
Inkling is a multimodal Mixture-of-Experts model from Thinking Machines Lab. It accepts text, image, video, and audio inputs and generates text.
Architecture
Inkling uses a 66-layer decoder with hybrid local and global attention. Each sparse feed-forward layer routes a token to 6 of 256 experts and also evaluates 2 shared experts. Images and video use a hierarchical patch encoder, while audio uses discrete dMel tokens.
NeMo AutoModel retains the checkpoint’s fused expert layout and supports pipeline parallelism and expert parallelism for full-parameter fine-tuning.
Example HF Models
Example Recipes
The model requires a multi-node launch. Adjust the data, pipeline, and expert-parallel dimensions to the available cluster before running the recipe.