Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting

Accurate stereo egocentric 3D pose by lifting joint heatmaps with Grid ViT attention and skeleton-aware propagation.

Taeho Kang1, Youngki Lee1
1Seoul National University
CVPR 2024 Highlight
Paper Official Supplementary Code arXiv

Abstract

We present EgoTAP, a heatmap-to-3D pose lifting method for highly accurate stereo egocentric 3D pose estimation. Severe self-occlusion and out-of-view limbs make accurate pose estimation challenging. Prior methods use joint heatmaps, but heatmap-to-3D conversion remains inaccurate.

We propose a Grid ViT Encoder that summarizes joint heatmaps into effective embeddings via self-attention, and a Propagation Network that uses skeletal structure to estimate obscure joints. EgoTAP reduces MPJPE by 23.9% vs. previous state of the art.

Method

Prior heatmap-to-3D lifters compress stereo heatmaps too aggressively and then guess missing limbs from full-body priors, which fails under heavy self-occlusion. EgoTAP splits the problem into two stages that preserve 2D evidence and inject skeletal structure only where it helps.

A Grid ViT Encoder tiles joint heatmaps into patches and applies multi-head self-attention so every joint embedding can still use global spatial context. A Propagation Network then walks the skeleton from the camera-attached root toward the extremities: each Propagation Unit blends parent and child features by joint certainty, recovering limbs that are weak or invisible in the input heatmaps.

EgoTAP architecture

Heatmap Estimator → Grid ViT → Propagation Network → per-joint projection.

Results

We compare against stereo egocentric baselines on the synthetic UnrealEgo benchmark and real EgoCap captures. In the clips, red is the prediction and blue is ground truth. EgoTAP reduces UnrealEgo MPJPE by 23.9% relative to the previous state of the art, with the largest gains on occluded and extreme poses.

UnrealEgo

EgoCap

Method UnrealEgo MPJPE (PA) EgoCap MPJPE (PA)
EgoGlass81.55 (61.56)67.90 (—)
UnrealEgo63.53 (47.76)70.77 (52.91)
Ego3DPose53.99 (43.02)69.45 (49.98)
EgoTAP (Ours)41.06 (35.39)55.38 (45.24)

BibTeX

@InProceedings{kang2024egotap,
  author    = {Kang, Taeho and Lee, Youngki},
  title     = {Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {842--851}
}