Collaboration with Knoxlabs

We’re happy to announce our first official collaboration here in Los Angeles. Knoxlabs is a leading distributor and integration partner of robotics solutions. Taron, Robert and their team have years of expertise in sourcing the latest and greatest robotics equipment from a wide range of suppliers. On top of their sourcing experience they are also an they also help integrate components into a working system, backed by hands-on engineering experience, not only by finding the best components, but also by providing the knowledge how to make them work together in the best way.

Xikaku is building hardware and software products to revolutionize robotic training and teleoperation. Starting with our deep knowledge of inertial measurement units and mixed reality headset tracking, we’re now applying our skills to provide motion capture solutions to gather the best possible data for training language-action models. With the humanoid robotics industry in the US rapidly evolving we’re looking to provide the tools to finally bring humanoid robots into the places where they’re needed most. Who could be a better partner to embark on this journey together than Knoxlabs!

As a first step we’ll soon launch the Xicap tracker. The Xicap tracker is a wrist worn multi-purpose tracker that can either be tracked visually as an Apple Vision Pro accessory or via electromagnetic tracking. This device will be offered by Knoxlabs as premium distribution and integration partner.

An extension to Xicap tracker will be Xicap glove, a touch sensitive motion capture glove that focuses on the tactile force measurements while its wearer manipulates objects. Stay tuned for updates!

Introducing Xicap - A Spatial Accessory for the Vision Pro

Portable outside-in tracking

It’s been a few months since Apple introduced active spatial accessories for the Vision Pro. Nathan Kong made a very nice presentation showing how these are made, how they are used and how they are integrated with the Vision Pro’s API. Spatial tags for the Vision Pro have existed for a while, they are simply QR code markers that get picked up by the AVP’s cameras.

Active spatial markers have the advantage that they can be tracked with higher accuracy - especially orientation - and they’re less prone to partial occlusions.

Active spatial tags in principle are similar to QR codes, they show a unique, known visual pattern defined by infrared light LEDs to the headset’s multiple cameras that allows the Vision Pro’s software to reconstruct the pose of the marker using some type of PnP algorithm.

Optical targets like these are quite common in human motion capture, outside-in tracking systems like ART have been using them almost for decades. We ourselves are using such camera systems frequently to do headset pose tracking either in multi-user setups or for in-vehicle motion capture.

What is different in case of the Vision Pro is that instead of placing cameras in static locations to define the tracking volume, the spatial accessory markers are being tracked by the cameras built into the Vision Pro. The user is taking the tracking volume with them.

The Xicap tracker is available for pre-order now.

Xicap spatial multi-purpose tracker

The Xicap tracker is a configurable multi-purpose spatial accessory for the Vision Pro. It is compact and has a rounded shape that can be easily attached to an object (e.g. for tool tracking, as shown in the image above) or be worn on the body (e.g. as game controller or motion capture device). We will offer several snap-on shells that for example allow the tracker to be attached to an object using a standard 1/4“-20 UNC screw. We will also offer straps to attach the tracker to arms, wrists or legs.

The image at the top of this post shows the Xicap tracker. To be as versatile as possible, the tracker has two push buttons on the side for users to access custom functionality. Additionally to the push buttons, the unit also has a multi-purpose IO connector to connect switches and other peripherals that could extend the functionality of the tracker - think haptic controls for custom game controllers.

The Xicap tracker is available for pre-order now.

Applications from tool tracking to robot teleoperation

There’s a wide range of applications for a spatial accessory like Xicap. Two areas specifically come to mind. As previously mentioned, accurately tracking tools during a critical maintenance task e.g. in the defense field and therefore being able to provide enhanced guidance to maintenance personnel via the Vision Pro can have tremendous value by enhancing efficiency and safety.

As the whole world is talking about AI in the form of large language models, in San Francisco, Shenzhen and Munich, the age of physical artificial intelligence has started. Humanoid robots are being trained to perform benign household tasks, such as folding the laundry. While for training LLMs scientists just dumped the whole internet into their training clusters, gathering reliable data to train robotic models is a lot harder. Human motion capture based on trackers such as Xicap plays an important role here.

Of course apart from these two evolving application areas there are no limits to what you can do with the Xicap tracker. It’s made to explore and experiment!

The Xicap tracker is available for pre-order now.

The specs

Here’s a quick look at what Xicap brings along. The first production version is still being finalized, so treat these as close-but-not-final:

The Xicap tracker is available for pre-order now.

Give us feedback

You probably noticed that we didn’t show any actual photos and videos of the actual Xicap unit in action yet. That is because we’re still working on finalizing the product. Right now the first PCBs of the electronics board of the unit are under production. Our designers are busy 3D printing to figure out the optimum shape for the shell, while the firmware engineers are finalizing Xicap’s MCU code. We will keep you updated on the timeline but expect the first product version to be ready at the end of October.

You’re welcome to drop us a message to sign up for updates or with questions, ideas or feedback here.

Physical AI with FusionHub - Part 1

There’s hardly any field where AI frontier models aren’t being used for generating code, creating marketing content, developing data analyses and many other applications. This happens mostly asynchronously though, the real-time processing of physical data has been a place where classic filters (eg. Kalman filters) or machine learning algorithms are applied to solve problems in areas such as pattern recognition and anomaly detection.

Then where do large language models (LLM) fit into a real-time sensor fusion system? Not where you might first expect. In this post - using FusionHub - we give a reasoning model direct access to a live sensor processing pipeline. We think that this combination is what physical AI can look like in practice and what could be applicable to a wide range of industrial monitoring and control use-cases.

In this article I’m describing a simple example of how to connect purpose-optimized real-time data processing with the vast general knowledge of an LLM. While this process could also be done manually by recording data from a real-time system and then feeding it into an LLM separately, with FusionHub we provide a platform where we can do this in a streamlined or even agentic way.

There are plenty of real world applications for such systems. The title image of this post shows a factory floor with various sensors attached to machines. Data from each sensor is preprocessed in real-time, eg. extracting the frequency spectrum from time-series data via FFT, and this data is then autmatically be passed on to a frontier model for a “bigger picture” data analysis, guiding a plant operator or maintenance staff to points in a factory that need inspection.

Reflexes and Reasoning

A sensor fusion pipeline has hard real-time obligations. An IMU delivers data at hundreds of hertz, a filter integrates every sample, and the fused output has to arrive with millisecond latency, deterministically, every single cycle. The processing times of a language model would be much too slow to work in this loop. The diagram below shows our apporach how the slow processing cycles of an LLm can be connected to fast-paced real-time analysis loop.

One might think of it as reflexes and reasoning. The fast loop - filtering, fusion, coordinate transforms - stays classic: a deterministic algorithms, tuned and verified, running at full rate. The LLM operates on a slow loop above it, on a timescale of seconds. It configures the fast loop, watches it, diagnoses it and adapts it. It does the things that previously required a human engineer with a terminal window and lots of domain knowledge.

That division of labor is our core idea. Everything below is an application of it.

Classifying Sensor Data

As a simple example of this technology, in order to classify the motion of one of our LPMS-IG1 inertial measurement units, we first created a minimal node graph where we send the output from the IMU to a data monitor to plot a time series graph. This gives us very clear information on what the different data values of the IMU look like on a time scale, our reflex pipeline so to say, but it allows us very little insight into what’s actually happening to the sensor without taking a deeper look at the data.

This is where the reasoning aspect of our system comes into play. By giving a large language model the tools to operate freely on our node graph to inspect data as well as giving it prior knowledge about how a node graph works and what kind of nodes exist, we enable it to use its vast built-in knowledge to make a zero-shot well-informed guess of what kind of motion we’re looking at.

The result above shows how the LLM first autonomously inspected the node graph, second extracted a window of live data from the graph and finally drew a quite accurate conclusion on the meaning of the acquired data.

Giving the Model Hands

FusionHub’s processing pipeline is defined by a node graph: sources (sensors, network inputs), filters (fusion, transforms, scripts) and sinks (network outputs, loggers, visualization). The graph the user wires in the editor is not a picture of the runtime - it is the runtime. Every edge corresponds to a real data subscription in the engine.

The AI Monitor built into FusionHub connects a language model to exactly this graph. The model gets a set of tools that mirror what a human operator can do:

• Inspect: read the pipeline configuration, list all available node types with their settings schemas, check per-node status and data rates, search the backend logs.

• Observe: subscribe to any node’s live output and pull actual data samples - fused poses, raw IMU frames, GNSS fixes - straight from the running pipeline.

• Act: add and remove nodes, connect them, and change node settings - including live updates.

• Learn: read FusionHub’s full documentation, answers about a node’s behavior come from the actual code and manual, not from model memory.

Two properties make this trustworthy rather than magical. First, the model works with the same primitive the human does: the node graph. When the AI adds a filter, the filter appears in the editor, and you can inspect, modify or delete it like any node you placed yourself. Second, every tool call is shown live in the UI as it happens - you watch the model read the log, probe a node, change a setting. There is no hidden channel.

Keep the Human in the Loop

The example shown here is not intended to be proof that a classification of real-time data acquired from the node graph by an LLM is under all circumstances true and reliable. LLMs are notorious for misclassifications if confronted with ambiguous data or incomplete prompts - as are humans, one might admit. The purpose of the FusionHub AI tooling, though, is to provide an extended method for engineers to make parameter optimizations faster and more efficient. This gives FusionHub abilities that no other real-time data processing tool on the market has.

This concludes the first part of our Phsyical AI with FusionHub series. We’ll be back soon with an example demonstrating the full inspect, observe, act and learn cycle. In the meantime explore our inertial measurement units or download FusionHub.