ComfyUI Workflow Builder

Translates natural language requests into executable ComfyUI workflow JSON. Always validates against inventory before generating.

Workflow Generation Process

Step 1: Understand the Request

Parse the user's intent into:

Output type

Image, video, or audio

Source material

Text-only, reference image(s), existing video

Identity method

None, zero-shot (InstantID/PuLID), LoRA, Kontext

Quality level

Draft (fast iteration) vs production (maximum quality)

Special requirements

ControlNet, inpainting, upscaling, lip-sync

Step 2: Check Inventory

Read

state/inventory.json

to determine:

Available checkpoints → select best match for task

Available identity models → determine which methods are possible

Available ControlNet models → enable pose/depth control if available

Custom nodes installed → verify all required nodes exist

VRAM available → optimize settings accordingly

Step 3: Select Pipeline Pattern

Based on request + inventory, choose from:

Pattern

When

Key Nodes

Text-to-Image

Simple generation

Checkpoint → CLIP → KSampler → VAE

Identity-Preserved Image

Character consistency

+ InstantID/PuLID/IP-Adapter

LoRA Character

Trained character

+ LoRA Loader

Image-to-Video (Wan)

High-quality video

Diffusion Model → Wan I2V → Video Combine

Image-to-Video (AnimateDiff)

Fast video, motion control

+ AnimateDiff Loader + Motion LoRAs

Talking Head

Character speaks

Image → Video → Voice → Lip-Sync

Upscale

Enhance resolution

Image → UltimateSDUpscale → Save

Inpainting

Edit regions

Image + Mask → Inpaint Model → KSampler

Step 4: Generate Workflow JSON

ComfyUI workflow format:

{

"{node_id}"

:

{

"class_type"

:

"{NodeClassName}"

,

"inputs"

:

{

"{param_name}"

:

"{value}"

,

"{connected_param}"

:

[

"{source_node_id}"

,

{

output_index

}

]

}

}

}

Rules:

Node IDs are strings (typically "1", "2", "3"...)

Connected inputs use array format:

["source_node_id", output_index]

Output index is 0-based integer

Filenames must match exactly what's in inventory

Seed values: use random large integer or fixed for reproducibility

Step 5: Validate

Before presenting to user:

Every

class_type

exists in inventory's node list

Every model filename exists in inventory's model list

All required connections are present (no dangling inputs)

VRAM estimate doesn't exceed available VRAM

Resolution is compatible with chosen model (512 for SD1.5, 1024 for SDXL/FLUX)

Step 6: Output

If online mode

Queue via

comfyui-api

skill

If offline mode

Save JSON to

projects/{project}/workflows/

with descriptive name

Workflow Templates

Basic Text-to-Image (FLUX)

{

"1"

:

{

"class_type"

:

"LoadCheckpoint"

,

"inputs"

:

{

"ckpt_name"

:

"flux1-dev.safetensors"

}

}

,

"2"

:

{

"class_type"

:

"CLIPTextEncode"

,

"inputs"

:

{

"text"

:

"{positive_prompt}"

,

"clip"

:

[

"1"

,

1

]

}

}

,

"3"

:

{

"class_type"

:

"CLIPTextEncode"

,

"inputs"

:

{

"text"

:

"{negative_prompt}"

,

"clip"

:

[

"1"

,

1

]

}

}

,

"4"

:

{

"class_type"

:

"EmptyLatentImage"

,

"inputs"

:

{

"width"

:

1024

,

"height"

:

1024

,

"batch_size"

:

1

}

}

,

"5"

:

{

"class_type"

:

"KSampler"

,

"inputs"

:

{

"seed"

:

42

,

"steps"

:

25

,

"cfg"

:

3.5

,

"sampler_name"

:

"euler"

,

"scheduler"

:

"normal"

,

"denoise"

:

1.0

,

"model"

:

[

"1"

,

0

]

,

"positive"

:

[

"2"

,

0

]

,

"negative"

:

[

"3"

,

0

]

,

"latent_image"

:

[

"4"

,

0

]

}

}

,

"6"

:

{

"class_type"

:

"VAEDecode"

,

"inputs"

:

{

"samples"

:

[

"5"

,

0

]

,

"vae"

:

[

"1"

,

2

]

}

}

,

"7"

:

{

"class_type"

:

"SaveImage"

,

"inputs"

:

{

"filename_prefix"

:

"output"

,

"images"

:

[

"6"

,

0

]

}

}

}

With Identity Preservation (InstantID + IP-Adapter)

Extends basic template by adding:

Load reference image node

InstantID Model Loader + Apply InstantID

IPAdapter Unified Loader + Apply IPAdapter

FaceDetailer post-processing

See

references/workflows.md

for complete node settings.

Video Generation (Wan I2V)

Uses different loader chain:

Load Diffusion Model (not LoadCheckpoint)

Wan I2V Conditioning

EmptySD3LatentImage (with frame count)

Video Combine (VHS)

See

references/workflows.md

Workflow 4 for complete settings.

VRAM Estimation

Component

Approximate VRAM

FLUX FP16

16GB

FLUX FP8

8GB

SDXL

6GB

SD1.5

4GB

InstantID

+4GB

IP-Adapter

+2GB

ControlNet (each)

+1.5GB

Wan 14B

20GB

Wan 1.3B

5GB

AnimateDiff

+3GB

FaceDetailer

+2GB

Common Mistakes to Avoid

Wrong output index

CheckpointLoader outputs

[model, clip, vae]

at indices

[0, 1, 2]

CFG too high for InstantID

Use 4-5, not default 7-8

Wrong resolution for model

FLUX/SDXL=1024, SD1.5=512

Missing VAE

FLUX needs explicit VAE (

ae.safetensors

)

Wrong model in wrong loader

Diffusion models need LoadDiffusionModel , not LoadCheckpoint Reference Files references/workflows.md - Detailed node-by-node templates references/models.md - Model files and paths references/prompt-templates.md - Model-specific prompts state/inventory.json - Current inventory cache