m2C2: mmWave-Camera Calibration via Human Pose

m2C2 estimates the rigid transformation \(T = \{R, \mathbf{t}\}\) that maps points from the mmWave radar frame to the camera frame. Instead of matching raw radar points to pixels, both sensors are first converted into the same representation: the joints of an SMPL human skeleton. The joint streams are aligned in time, and the extrinsics are then solved by minimising a confidence-weighted reprojection error while jointly optimising per-bone length scale factors. Since the method only requires human pose, it can be applied to any sensor pair that supports SMPL pose estimation.

3D joint positions \(\mathbf{p}^{\mathcal{M}}_{j,k}\) are estimated from the radar point cloud with mmMesh, and mmJoints adds a per-joint confidence \(w^{\mathcal{M}}_{j,k}\). On the camera side, HybrIK provides 2D joint positions \(\mathbf{p}^{\mathcal{C}}_{j,k}\), with a confidence \(w^{\mathcal{C}}_{j,k}\) taken from the heatmap peak. Both outputs are truncated to the same \(N = 22\) joints so every radar joint has a one-to-one partner in the image.

The two sensors have different latencies, leaving a sub-frame offset \(\Delta\tau\) between the streams. Joint positions cannot be compared directly across 3D and 2D, but joint speeds correlate well once the streams are aligned. We therefore compute a motion signature from the median speed of six robust joints \(\mathcal{J}\) (pelvis, ankles, neck and shoulders), normalise it with the median absolute deviation, and find the offset that maximises the continuous cross-correlation:

\[ \sigma[k] = \operatorname*{median}_{j \in \mathcal{J}} \left\| \mathbf{p}_{j,k} - \mathbf{p}_{j,k-1} \right\|_2, \qquad \Delta\tau^{*} = \operatorname*{arg\,max}_{\Delta\tau \in [-w,\, w]} \sum_{k} \tilde{\sigma}^{\text{3D}}(\tau_k)\, \tilde{\sigma}^{\text{2D}}(\tau_k + \Delta\tau) \]

The radar stream is then resampled at the shifted camera timestamps using piecewise cubic Hermite interpolation (PCHIP).

mmMesh and HybrIK estimate skeletons with systematically different proportions. If left uncorrected, the optimiser compensates for this by moving the camera, which biases the extrinsics. m2C2 instead rescales each bone along the kinematic chain with a learnable factor \(s_j = \exp(\lambda_j)\), which changes bone lengths while preserving joint angles. The scale factors are optimised jointly with \(T\), with a regulariser that keeps them close to the estimator's original lengths:

\[ \hat{\mathbf{p}}_j = \hat{\mathbf{p}}_{\mathrm{pa}(j)} + s_j \left( \mathbf{p}_j - \mathbf{p}_{\mathrm{pa}(j)} \right), \qquad T^{*}, \{\lambda_j^{*}\} = \operatorname*{arg\,min}_{T,\, \{\lambda_j\}} \; \mathcal{L}_{\text{reproj}} + \lambda_{\text{reg}} \sum_{j=1}^{N} \lambda_j^{2} \]