Architecture
Hierarchical Motion Transformer
A spatial-to-temporal model hierarchy: joint-graph attention over a 21-joint skeleton, ~11M parameters, 100-frame windows at 25 Hz, dual outputs — tokens and a 384-d embedding.
A hierarchy: space, then time
The Hierarchical Motion Transformer reads movement the way an analyst does: one body at a time, then across time. Each input is a 100-frame window — four seconds of motion at 25 Hz — over the 21-joint skeleton. Spatial attention layers first resolve how joints relate within a single instant: how the knee loads, how the trunk counter-rotates, how the arms balance a cut. Temporal layers then model how those poses evolve — from sub-second micro-adjustments to the full arc of a sprint.
Factoring space and time keeps the model compact — roughly 11 million parameters — without flattening the structure of the body into a generic token soup. The skeleton stays a skeleton all the way through the network.
Attention that knows the body
Plain transformers treat inputs as an unordered set; bodies are not. Joint-graph attention biases every attention score with a learned term indexed by hop distance — how many bones separate two joints on the skeleton graph. A wrist attends to an elbow across one hop; to the opposite ankle across many. The prior is soft: the model can still discover long-range couplings — arm swing against stride, trunk lean against deceleration — but it starts from anatomy instead of rediscovering it.
The 21-joint, 20-bone topology this bias is computed over is documented in the skeleton reference.
Two outputs, one pass
Every forward pass produces two views of the same window. Per-joint tokens carry local detail for reconstruction work — denoising, inpainting, 2D→3D lifting. A single 384-dimensional embedding summarizes the whole sequence for retrieval, clustering, and probing. Downstream products consume one or both; nothing needs a second model.
How these outputs are trained is covered in foundation-model training; how they are measured, in the evaluation suite.
Model datasheet
| Parameter | Value | Note |
|---|---|---|
| backbone | Hierarchical Motion Transformer | spatial → temporal factorization |
| parameters | ~11M | compact by design — fast to iterate, cheap to serve |
| input window | 100 frames · 4 s | sampled at 25 Hz |
| skeleton | 21 joints · 20 bones | documented in the skeleton reference |
| attention bias | hop-distance relative position | computed on the skeleton graph |
| outputs | per-joint tokens + 384-d embedding | one forward pass, two views |
| objectives | masked reconstruction + multi-view consistency + physics terms | see foundation-model training |