Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting

Stereo egocentric methods already have usable joint heatmaps; the accuracy is lost when those heatmaps are lifted to a 3D pose.

VR and AR want a body pose from the sensors a headset already carries, so the camera looks down at the body from one of the hardest viewpoints: limbs leave the frame, the body occludes itself, and the fisheye lens distorts what remains. Direct 2D joint estimation is unreliable here, so egocentric methods predict joint heatmaps, a distribution over where each joint might be.

The common pipeline compresses all the heatmaps into one embedding with a CNN encoder and lifts it to the 3D pose with a multilayer perceptron. Two things go wrong there.

The first is the encoder. On the left are four joint heatmaps, hand, elbow, knee and foot, each with its own peak. The CNN encoder merges them into one shared embedding, the single column in the middle, and on the right a decoder recovers the same blur for every joint: which heatmap belonged to which joint is lost.

Convolution also assumes related things sit near each other in the image, and the egocentric view breaks that. On the left the blue limb links shoulder to hand; in the camera image on the right the shoulder sits at the top and the hand near the bottom, so the convolution window covering the shoulder never sees the hand.

The second is the lifting step. The multilayer perceptron maps the embedding to the full body with no notion of how certain each joint is or how the joints connect, so a feature from an obscure joint can corrupt a joint with clear visual cues.

Keep one embedding per heatmap so no joint loses its identity, and let the joints the camera sees clearly predict the ones it cannot.

Swapping the CNN for a transformer is not by itself the fix: a ViT that stacks the heatmaps along the channel axis before patching again loses which embedding belongs to which joint. Each heatmap needs its own patches, with self-attention running across all of them, so every joint keeps an embedding and distant joints can still communicate.

The second idea rests on two properties of the setup: a joint both stereo cameras see has enough in its own two heatmaps for a precise 3D position, and joints nearer the camera tend to be more visible than those farther away. So estimation runs outward along the skeleton, from neck to extremities, and a child joint inherits its parent’s estimate only when its own evidence is weak.

EgoTAP is a Grid ViT Heatmap Encoder followed by a Propagation Network, both on top of a frozen heatmap estimator.

The pre-trained estimator outputs one heatmap per joint per view plus limb heatmaps for the segments between joints; only the lifting is trained.

On the left, the joint heatmaps are tiled into one grid image, one heatmap per cell, and patched. The blue double arrow joining two opposite corners marks that self-attention is global: a joint’s patches can draw on a distant joint’s wherever it lies in the image. Each input patch stays paired with its output, so the outputs for one heatmap are re-encoded into that heatmap’s embedding, and the two views’ embeddings for a joint form its joint feature F_J, the columns on the right, one per joint. The colour columns are illustrative, projected from the heatmaps rather than read out of the trained encoder.

On the left, propagation runs outward along the body: filled circles are visible joints, the hollow circle is the obscure hand, and the arrows run shoulder to elbow to hand, the step this unit computes. The Propagation Unit is an LSTM-inspired cell. It takes the parent’s hidden state h_parent, here the elbow’s, the relational feature F_R from the elbow–hand limb heatmap, and the child’s own joint feature F_J. Before the LSTM cell, an extra forget gate σ, computed from the child’s joint feature, scales the inherited state and the relational feature: evident child features forget the inheritance, obscure ones keep it. The output is the propagated feature F_P, concatenated with F_J and projected to the joint’s 3D position.

EgoTAP works from a single stereo frame and only from what is visible.

Motions with severe occlusion, Crawling and Sitting on the Ground, keep a much higher error than the other categories, and Crawling is also where the gain over prior work is smallest. The Propagation Network assumes the stereo setup, where a joint visible in both views fixes its own 3D position, so a monocular version would need modification, and it follows a fixed tree from the head rather than a learned graph.