Physical AI with FusionHub - Part 1

There’s hardly any field where AI frontier models aren’t being used for generating code, creating marketing content, developing data analyses and many other applications. This happens mostly asynchronously though, the real-time processing of physical data has been a place where classic filter or machine learning algorithms are applied to solve problems in areas such as pattern recognition and anomaly detection.

Then where do large language models (LLM) fit into a real-time sensor fusion system? Not where you might first expect. In this post - using FusionHub - we give a reasoning model direct access to a live sensor processing pipeline. We think that this combination is what physical AI can look like in practice and what could be applicable to a wide range of industrial monitoring and control use-cases.

Reflexes and Reasoning

A sensor fusion pipeline has hard real-time obligations. An IMU delivers data at hundreds of hertz, a filter integrates every sample, and the fused output has to arrive with millisecond latency, deterministically, every single cycle. The processing times of a language model would be much too slow to work in this loop. The diagram below shows our apporach how the slow processing cycles of an LLm can be connected to fast-paced real-time analysis loop.

One might think of it as reflexes and reasoning. The fast loop - filtering, fusion, coordinate transforms - stays classic: a deterministic algorithms, tuned and verified, running at full rate. The LLM operates on a slow loop above it, on a timescale of seconds. It configures the fast loop, watches it, diagnoses it and adapts it. It does the things that previously required a human engineer with a terminal window and lots of domain knowledge.

That division of labor is the core idea. Everything below is an application of it.

Classifying Sensor Data

As a simple exmaple of this technique, in order to classify the motion of one of our LPMS-IG1 inertial measurement units, we first created a minimal node graph where we send the output from the IMU to a data monitor to plot a time series graph. This gives us very clear information on what the different data values of the IMU look like on a time scale, our reflex pipeline so to say, but it allows us very little insight into what’s actually happening to the sensor without taking a deeper look at the data.

This is where the reasoning aspect of our system comes into play. By giving a large language model the tools to operate freely on our node graph to inspect data as well as giving it prior knowledge about how a node graph works and what kind of nodes exist, we enable it to use it’s vast built-in knowledge to make a zero-shot well-informed guess of what kind of motion we’re looking at.

The result above shows how the LLM first autonomously inspected the node graph, second extracted a window of live data from the graph and finally drew a quite accurate conclusion on the meaning of the acquired data.

Giving the Model Hands

FusionHub’s processing pipeline is defined by a node graph: sources (sensors, network inputs), filters (fusion, transforms, scripts) and sinks (network outputs, loggers, visualization). The graph the user wires in the editor is not a picture of the runtime - it is the runtime. Every edge corresponds to a real data subscription in the engine.

The AI Monitor built into FusionHub connects a language model to exactly this graph. The model gets a set of tools that mirror what a human operator can do:

• Inspect: read the pipeline configuration, list all available node types with their settings schemas, check per-node status and data rates, search the backend logs.

• Observe: subscribe to any node’s live output and pull actual data samples - fused poses, raw IMU frames, GNSS fixes - straight from the running pipeline.

• Act: add and remove nodes, connect them, and change node settings - including live updates.

• Learn: read FusionHub’s full documentation, answers about a node’s behavior come from the actual code and manual, not from model memory.

Two properties make this trustworthy rather than magical. First, the model works with the same primitive the human does: the node graph. When the AI adds a filter, the filter appears in the editor, and you can inspect, modify or delete it like any node you placed yourself. Second, every tool call is shown live in the UI as it happens - you watch the model read the log, probe a node, change a setting. There is no hidden channel.

Keep the Human in the Loop

The example shown here is not intended to be proof that a classification of real-time data acquired from the node graph by an LLM is under all circumstances true or reliable. LLMs are notorious for misclassifications if confronted with ambiguous data or incomplete prompts, as are humans one might admit. The purpose of the FusionHub AI tooling, though, is to provide an extended method for engineers to make parameter optimizations faster and more efficient. This gives FusionHub abilities that no other real-time data processing tool on the market has.

This already concludes the first part of our Phsyical AI with FusionHub series. We’ll be back soon with an example demonstrating the full inspect, observe, act and learn cycle. In the meantime explore our inertial measurement units or download FusionHub.