Capturing Motion for Dexterous Manipulation Tasks

Recording training data for robotics model fine tuning might just as well be the killer app for the Vision Pro. It has it all: excellent finger tracking, great pass through, good camera feed to route back to the model for training.

For model tuning three kinds of training data are important: synthetic, mocap and teleop. We want to cover all three with the same workflow. So far sim and MR mocap, real robot soon!

Working towards a fully integrated data capture pipeline for robot training, seamless switching between mixed reality, simulation and (soon) teleop is needed. Using our FusionHub backend this was quite straightforward to implement. Check the node graph in the top left.