Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views

Binocular egocentric 3D pose that uses stereo correspondence and perspective, not only 2D appearance.

Taeho Kang1, Kyungjin Lee1, Jinrui Zhang2, Youngki Lee1
1Seoul National University    2Central South University
SIGGRAPH Asia 2023
Paper Official Supplementary Code arXiv

Abstract

We present Ego3DPose, a highly accurate binocular egocentric 3D pose reconstruction system. Severe viewing distortion, self-occlusion, and limited field-of-view make this setup challenging. We observe that two important 3D cues — stereo correspondences and perspective — are underused by prior work that relies mainly on 2D image features.

We propose a two-path network that estimates pose per limb independently from binocular heatmaps, reducing bias toward trained full-body distributions, and a perspective-aware trigonometry representation for limb 3D orientation. An end-to-end system combining both techniques reduces MPJPE by 23.1% vs. prior SOTA on UnrealEgo.

Method

Egocentric fisheye views make full-body 2D features unreliable: limbs are distorted, often self-occluded, and sometimes leave the field of view. Ego3DPose therefore estimates pose per limb from binocular cues instead of forcing a single body-level prior.

Optical features include joint-position heatmaps and Perspective Embedding Heatmaps, which encode limb orientation with trigonometry of the viewing angle. A Stereo Matcher recovers 3D limb orientations from the stereo pair without requiring full-body context; a pose decoder then assembles the final 3D skeleton.

Ego3DPose architecture

Optical features → two-path aggregation → 3D pose reconstruction.

Results

Comparisons on synthetic UnrealEgo and real EgoCap. Red is ground truth and gray is the prediction. Combined with Perspective Embedding Heatmaps, Ego3DPose cuts UnrealEgo MPJPE by 23.1% vs. prior state of the art.

UnrealEgo

EgoCap

Method UnrealEgo MPJPE / PA EgoCap MPJPE / PA
EgoGlass83.33 / 61.5661.78 / 46.06
UnrealEgo79.08 / 59.2659.16 / 48.22
Ego3DPose (Ours)60.82 / 48.4754.41 / 40.24

BibTeX

@inproceedings{10.1145/3610548.3618147,
  author    = {Kang, Taeho and Lee, Kyungjin and Zhang, Jinrui and Lee, Youngki},
  title     = {Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views},
  year      = {2023},
  month     = {December},
  isbn      = {9798400703157},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  url       = {https://doi.org/10.1145/3610548.3618147},
  doi       = {10.1145/3610548.3618147},
  booktitle = {SIGGRAPH Asia 2023 Conference Papers},
  pages     = {1--10},
  articleno = {82},
  series    = {SA '23}
}