ECCV 2026

DSAR Dual-Stream Autoregressive Modeling of Temporal Cloth Dynamics for Photorealistic Animatable Avatars

Haozhong Xiong1 Yao Yu1,* Yu Zhou1,* Sidan Du1
1 Nanjing University, Nanjing, China * Corresponding authors
Four photorealistic animatable avatars wearing loose garments shown in different body poses.
DSAR reconstructs photorealistic animatable avatars while preserving momentum-driven garment deformation across loose clothing and unseen motion sequences.

Abstract

Creating photorealistic, temporally coherent animatable avatars from RGB video is difficult because the current skeletal pose does not uniquely determine garment state. Pose-conditioned models therefore tend to over-smooth cloth appearance, flicker over time, or fail on motion outside the training distribution. DSAR treats cloth as a causal temporal process with two complementary autoregressive streams: a geometric stream propagates the previous frame's predicted surface deformation, while a state stream retrieves historical temporal states from a memory bank. Motion-Adaptive Temporal Aggregation and motion-aware regularization accommodate spatially varying dynamics. Trained from multi-view photometric supervision, the framework improves rendering quality, temporal consistency, and generalization to unseen motion patterns.

Motivation

The same standing pose can follow a rapid spin or a period of rest, yet the corresponding garment states differ because momentum, wrinkles, tension, and settling behavior depend on motion history. Skeletal pose records body configuration but omits both the observable evolution of the cloth surface and latent internal state. A model that maps pose directly to appearance therefore learns an ambiguous correlation instead of the causal transition between cloth states.

Pose-state ambiguity

Nearly identical joint configurations can correspond to different garment shapes because cloth retains momentum and deformation history.

Missing surface kinematics

Skeletal input describes the body, but not the current cloth configuration, velocity, or directional momentum.

Missing internal state

Fabric tension, material response, and deformation memory are latent properties that cannot be recovered from pose alone.

Method

DSAR represents an avatar with a learnable canonical 3D Gaussian template bound to the SMPL-X surface through UV parameterization. At frame t, a StyleUNet processes a temporal window of pose-derived position and velocity maps together with the previous predicted cloth deformation. The geometric stream aggregates observable surface motion, while the state stream retrieves latent history from a memory bank. Their fused features predict frame-specific Gaussian deformations, which are transformed by Linear Blend Skinning and rendered through differentiable splatting.

DSAR pipeline with geometric and state streams, a motion encoder, MATA fusion, motion decoder, Gaussian template, linear blend skinning, and splatting.
Overview of DSAR. The geometric stream processes the temporal pose window and previous deformation through Motion-Adaptive Temporal Aggregation; the state stream fuses the resulting feature with retrieved memory states. A decoder predicts the current Gaussian deformation and updates the temporal state for subsequent frames.

Canonical Gaussian representation

An optimizable template is bound to the SMPL-X surface through UV parameterization. The network predicts frame-specific changes in Gaussian position, rotation, scale, opacity, and color.

Autoregressive geometric stream

Position maps, velocity maps, and the previous predicted deformation enter the StyleUNet encoder. This feedback explicitly propagates observable cloth configuration and motion across frames.

Memory-based state stream

A fixed-size memory bank stores recent temporal states. Cross-attention retrieves relevant history and fuses it with current geometric features to represent latent factors that geometry alone cannot observe.

Adaptive temporal regularization

Motion-dependent weights impose stronger consistency in static regions and relax feature, deformation, and trajectory regularization in fast-moving garment regions.

Motion-Adaptive Temporal Aggregation combining causal attention, a state memory bank, and cross-attention.

Motion-Adaptive Temporal Aggregation

At the deepest feature level, spatial motion magnitudes and temporal position embeddings condition causal self-attention over the input window. High-motion regions emphasize recent evidence for transient dynamics, while stable regions can retain broader context. Lightweight temporal convolutions aggregate the finer feature levels efficiently.

Results

DSAR is evaluated at 940×1280 resolution on held-out sequences from 4D-DRESS and AvatarREX, with AMASS motion used to test far-distribution generalization. Under identical training data, DSAR outperforms HumanNeRF, GaussianAvatar, and Animatable-GS across PSNR, SSIM, and LPIPS while preserving momentum-driven garment deformation on challenging motion.

Qualitative comparison of ground truth, DSAR, Animatable-GS, GS Avatar, and HumanNeRF on three people wearing loose clothing.
Qualitative comparison. On post-rotation stopping, arm motion, and jumping sequences, DSAR better preserves momentum-driven cloth evolution and garment structure; red boxes highlight regions where pose-conditioned baselines exhibit over-smoothing or artifacts.
PSNR ↑31.0765quantitative comparison
SSIM ↑0.9839quantitative comparison
LPIPS ↓0.0305quantitative comparison
Five reconstructed avatars shown across six body poses and diverse loose clothing styles.
Avatar gallery. Photorealistic reconstruction across diverse body shapes, viewpoints, loose garments, and motion patterns.
Ablation results comparing ground truth, full DSAR, and variants without adaptive regularization, geometric stream, or state stream.
Ablation study. Removing adaptive regularization, the geometric stream, or the state stream degrades reconstruction quality, with larger errors on far-distribution motion.

Demo

The supplementary video presents continuous animation sequences reconstructed from AvatarREX and 4D-DRESS. Loose jackets, dresses, and layered garments retain momentum, develop wrinkles, and settle over time instead of snapping to an appearance determined only by the current pose.