Run this workflow on InstaSD
Get started in minutes! Run this ComfyUI workflow online - no setup required.
Description
Unlock fluid style and motion transfer using WAN 2.1 FUN and Flux, with this innovative ComfyUI workflow. Designed for expressive and dynamic video generation, it enables seamless animation-to-animation transformations while preserving nuanced body movement and artistic styling.
π― Features
- Style & Motion Fusion β Combine two videos: one as style, the other as motion input.
- Supports WAN 2.1 FUN & Flux β Leverages cutting-edge video modeling architecture for optimal consistency.
- Fully ComfyUI Integrated β All nodes and preprocessing handled directly in ComfyUI, no external tools required.
- Dynamic Video Composition β Frame-by-frame synthesis with control over fps, resolution, and upscaling.
- Supports CFGZeroStar & DiT Models β Allows for advanced model patching and diffusion skipping techniques.
π‘ Use Cases
- Stylized Animation β Create fluid animated sequences with custom art styles.
- AI Music Videos β Transfer dance or movement from one video into a unique stylized output.
- Video-to-Video Transfer β Ideal for merging poses, expressions, or dynamics across different video clips.
- Motion Testing for Game Design β Rapidly prototype character movements with visual flair.
βοΈ How It Works
-
Prepare Your Inputs
- Choose a motion source video and a style source video.
- Ensure both are processed through
video_cn_preprocessnodes.
-
Load Essential Components
- Load WAN FUN 1.3B, WAN 2.1 VAE, and umt5_xxl_fp8 CLIP models.
- Include CLIP Vision, SkipLayerGuidanceDiT, and RIFE VFI nodes.
-
Apply Preprocessing & Alignment
- Use
DWPreprocessorand resolution normalization. - Encode visual style via
CLIPVisionEncodewithimg_restyle.
- Use
-
Combine Motion & Style
- Use
WanVideoTeaCacheKJto handle video-to-video transformation. - Utilize
ModelSamplingSD3andCFGZeroStarto optimize generation fidelity.
- Use
-
Postprocessing & Output
- Interpolate with
RIFE VFIfor smooth playback. - Export via
VHS_VideoCombinewith frame rate control, compression, and preview.
- Interpolate with
Credits: cgpixel23
Models
| File | Destination | Source |
|---|---|---|
| clip_vision_h.safetensors | /ComfyUI/models/clip_vision | Download |
| wan_2.1_vae.safetensors | /ComfyUI/models/vae | Download |
| umt5_xxl_fp8_e4m3fn_scaled.safetensors | /ComfyUI/models/clip_vision | Download |
| wan fun 1.3B.safetensors | /ComfyUI/models/unet | Download |
| yolox_l.onnx | /ComfyUI/custom_nodes/ComfyUI_controlnet_aux/ckpts/yzd-v/DWPose | Download |
| dw-ll_ucoco_384_bs5.torchscript.pt | /ComfyUI/custom_nodes/ComfyUI_controlnet_aux/ckpts/hr16/DWPose-TorchScript-BatchSize5 | Download |
| 4x-ClearRealityV1.pth | /ComfyUI/models/upscale_models | Download |
| clip_l.safetensors | /ComfyUI/models/clip | Download |
| flux.1-lite-8B-alpha.safetensors | /ComfyUI/models/unet | Download |
| depth_anything_vitl14.pth | /ComfyUI/custom_nodes/ComfyUI_controlnet_aux/ckpts/LiheYoung/Depth-Anything/checkpoints | Download |
| flux.1-lite-8B-alpha-Q5_K_S.gguf | /ComfyUI/models/unet | Download |
| flux1-depth-dev-lora.safetensors | /ComfyUI/models/loras | Download |
| flux1-canny-dev-lora.safetensors | /ComfyUI/models/loras | Download |
| turbo flux lora.safetensors | /ComfyUI/models/loras | Download |
| Studio_Ghibli_Flux.safetensors | /ComfyUI/models/loras | Download |
| ae.safetensors | /ComfyUI/models/vae | Download |
| umt5_xxl_fp8_e4m3fn_scaled.safetensors | /ComfyUI/models/clip | Download |
| t5xxl_fp8_e4m3fn.safetensors | /ComfyUI/models/clip | Download |
Nodes
SetNodeGetNodeVHS_VideoCombineImageListToImageBatchRIFE VFIImageConcatMultiCLIPVisionLoaderVAELoaderUNETLoaderSkipLayerGuidanceDiTWanVideoTeaCacheKJCLIPVisionEncodeDWPreprocessorINTConstantModelSamplingSD3CFGZeroStarVAEDecodeLayerUtility: PurgeVRAMVHS_VideoInfoLoadedImageResizeKJUpscaleModelLoaderImage Comparer (rgthree)PreviewImageeasy cleanGpuUsedImageScaleByImageUpscaleWithModelImageFromBatch+GetImageSize+SimpleMath+ImageScaleStringFunction|pysssssFlorence2ModelLoaderDualCLIPLoaderFlorence2RunCLIPTextEncodeKSamplerDepthAnythingPreprocessorDownloadAndLoadFlorence2ModelRerouteImageBlendSaveImageWanFunControlToVideoFast Groups Bypasser (rgthree)NoteUnetLoaderGGUFPower Lora Loader (rgthree)FluxGuidanceInstructPixToPixConditioningLoadImageCLIPLoaderVHS_LoadVideo