What I Learned at IROS: The Future of Robotics May Be About Imagination, Not Just Bigger Models

---

I spent today at IROS listening to researchers talk about foundation models, world models, energy-based models, and compositional planning for robots.

At first, these sounded like separate research topics. But after seeing the examples, I realized they are converging on a much bigger idea:

Perhaps the next leap in robotics will not come simply from training a bigger model to produce better actions. It will come from robots that can imagine possible futures, reason about them, compose different forms of intelligence, and only then act.

That sounds abstract. A simple example makes it much easier to understand.

The old question: “What should I do?”

Imagine telling a robot:

Put the white mug on the left plate and the yellow mug on the right plate.

A modern Vision-Language-Action model, or VLA, can broadly be thought of as learning:

what I see + what you asked me to do → what action I should take

So:

camera + instruction → neural network → robot action

This is enormously powerful. Models such as RT-2 helped establish the idea that web-scale vision-language knowledge could be connected to robotic actions.

But physical reality has an unpleasant property: small mistakes compound.

The robot moves slightly too low.

The gripper approaches the wrong part of the cup.

The cup has moved somewhere it wasn’t during training.

One mistake changes the world, which makes the next prediction harder. For a task lasting dozens of actions, this becomes increasingly problematic.

And unlike a chatbot, a robot cannot simply regenerate an answer.

It has already changed reality.

A different question: “What would happen if I did this?”

This was the first idea today that really clicked for me.

Instead of immediately executing an action, the robot can use a world model to predict what might happen.

Suppose the robot wants to grasp a mug.

It considers:

Action A: approach from the side.

Before actually moving, the world model generates the likely future:

The gripper may hit the rim.

So the robot considers:

Action B: approach slightly higher.

The imagined future now shows the gripper reaching the handle successfully.

Only then does it execute.

The recently proposed World Action Planner demonstrates exactly this type of architecture. A vision-language model proposes actions while an action-conditioned world model generates imagined rollouts. The system can then search over and refine those possibilities before executing them. The authors report generalization experiments involving new object layouts, compositional tasks and even settings without a learned expert robot policy.

This changes the nature of the robot.

Instead of:

Perceive → Act

we get:

Perceive → Propose → Imagine → Evaluate → Revise → Act

That is much closer to what we normally call planning.

A robot can “dream” before it moves

One of the most intuitive demonstrations I saw was Video Language Planning (VLP).

Imagine telling a robot:

Put the fruits in the top drawer.

There are many possible sequences:

Open drawer → put banana inside → put apple inside → close drawer

or:

Try putting banana inside → realize drawer is closed → fail

or perhaps:

Open drawer → put banana inside → accidentally close drawer too early → fail

Instead of discovering these mistakes physically, VLP uses generative video models to construct possible future sequences.

It then searches through these imagined futures.

The underlying 2024 ICLR work is quite interesting: VLP combines tree search with vision-language models acting as policies/value functions and video-generation models acting as dynamics models. More computation can therefore be spent exploring alternative video plans before execution.

In simplified form:

Current reality

↓

Imagine future A → bad

Imagine future B → promising

Imagine future C → bad

↓

Continue imagining B

↓

Find plausible path to goal

↓

Execute

Suddenly generative video isn’t merely about generating impressive videos.

It can become part of a robot’s reasoning machinery.

But who decides whether an imagined future is good?

This led to another idea from today’s talk that I particularly liked.

A Vision-Language Model can become a heuristic.

In classical AI, algorithms such as A* search don’t blindly explore every possibility. A heuristic estimates which states appear closer to the goal.

Now imagine doing something analogous with a foundation model.

Show a VLM an imagined video and tell it:

Goal: Put all the fruits in the drawer.

The VLM can assess whether that imagined future seems to be progressing toward the goal.

So a foundation model doesn’t necessarily have to control the robot.

It can instead play the role of:

1. critic

2. planner

3. goal evaluator

4. semantic reasoner

5. or search heuristic.

That distinction feels important.

We often talk about foundation models as if their ultimate destination is to swallow the entire robotics stack.

Perhaps that’s the wrong mental model.

Composition may matter as much as scale

This was probably my biggest takeaway from the session.

A robot has fundamentally different kinds of problems to solve.

Suppose it is installing a GPU into a computer.

At the semantic level it needs to understand:

“That is the GPU. That is the PCIe slot. The GPU needs to go there.”

Then it needs prediction:

“At this trajectory and angle, the GPU should line up with the slot.”

But when the GPU actually contacts the slot, the problem changes completely.

Now the robot needs to react to tiny physical signals:

“Force increased on the left. Correct the angle immediately.”

These are very different forms of intelligence.

And crucially, they operate at different timescales.

Language reasoning can be relatively slow.

Trajectory prediction needs to be faster.

Force and tactile control may need to react continuously.

Trying to make one giant model do all of these things at the same frequency may not always be desirable.

The CoStream idea

One architecture presented today, CoStream, makes this explicit.

It decomposes manipulation into three behaviors:

Semantic behavior — foundation models understand the scene and extract spatial constraints.

Predictive behavior — a video world model imagines motion and produces trajectory information.

Reactive behavior — tactile and force feedback provide high-frequency physical corrections.

These behaviors then compose into the robot’s control command.

So instead of:

one giant model → everything

we get something more like:

Reason -> Predict -> React

Then under React, we have:

• Compose

• Act

Even more interestingly, the components can run asynchronously.

The semantic brain doesn’t need to rethink the meaning of “insert the GPU” hundreds of times every second.

But the tactile controller may need to make rapid corrections while insertion is happening.

CoStream reports experiments across eight real-world tasks, including CPU, GPU and RAM assembly. Its project results highlight especially large gains in contact-rich assembly, as well as recovery from physical perturbations during execution.

This feels less like building one artificial brain and more like building an orchestra of specialized intelligence.

Interestingly, this isn’t entirely new

There is some historical irony here.

Robotics has long explored hierarchical control, behavior-based robotics, planning, model-predictive control and modular architectures.

Deep learning pushed the field strongly toward end-to-end learning:

observation → neural network → action

Now foundation models and generative models may be bringing some of those older architectural ideas back — but with dramatically more capable components.

The planner can understand natural language.

The perception system can recognize open-world objects.

The world model can generate possible futures.

The critic can reason semantically about those futures.

The low-level controller can still exploit decades of control theory.

So perhaps the future isn’t:

classical robotics OR foundation models.

It may be:

classical robotics + foundation models + world models + search + learned policies + control.

This looks surprisingly similar to AI agents

While watching the talk, another connection struck me.

Over the last few years, we’ve learned that a powerful LLM alone doesn’t necessarily make a powerful software agent.

Useful agents increasingly look like systems:

Model → Plan → Tool → Observe → Evaluate → Replan

The intelligence emerges partly from the loop and architecture around the model.

Robotics seems to be moving toward something remarkably similar:

Foundation model

↓

Plan

↓

World model imagination

↓

Evaluate

↓

Physical action

↓

Observe reality

↓

Replan

The difference is that the robot’s “tool environment” is the physical world.

And that difference matters.

A software agent that makes a bad API call can often retry.

A robot holding a glass cannot undo gravity.

From prediction to imagination

This leads to the idea I found most thought-provoking today.

Much of modern AI has been built around prediction.

LLMs predict tokens.

Vision models predict labels or representations.

Robot policies predict actions.

But intelligent behavior may require something beyond predicting the next action.

It requires asking:

“If I do this, what happens next?”

and eventually:

“Of all the futures I can imagine, which one best achieves my objective?”

That transforms a world model from a prediction engine into something resembling a simulation engine for decision-making.

Mathematically, the conceptual change is simple.

Instead of learning only:

{state} → {action}

we increasingly care about:

({state},{action}) → {future state}

because once I can model the second relationship, I can evaluate many actions without physically trying all of them.

That is enormously powerful in robotics.

And composition makes the space manageable

There is another reason composition matters.

Real-world possibilities explode combinatorially.

Imagine a robot that understands:

50 objects,

30 actions,

20 environments,

and 20 goals.

The possible combinations quickly become enormous.

We cannot realistically demonstrate every combination.

But perhaps the robot doesn’t need to see them all.

If it separately learns concepts such as:

inside

above

grasp

avoid

insert

push

then planning can potentially compose these capabilities into combinations that weren’t explicitly demonstrated.

This is the intuition behind compositional generalization.

Learn reusable pieces.

Then construct new behavior from them.

So will monolithic robot models disappear?

Probably not.

End-to-end VLA models have major advantages. They can learn extraordinarily rich behaviors directly from data without engineers specifying every intermediate representation.

And the results shown today shouldn’t be interpreted as proving that compositional architectures universally outperform end-to-end models. Benchmarks, training data, task distributions and hardware all matter.

The more interesting possibility is that the distinction itself eventually becomes less binary.

We may end up with large general-purpose models inside larger planning systems.

A VLA might provide powerful motor priors.

A VLM might provide semantic reasoning.

A world model might provide imagination.

Search might provide deliberation.

Tactile policies might provide reflexes.

Classical controllers might provide stability.

The important question becomes not:

“Which model controls the robot?”

but:

“How should all these forms of intelligence be composed?”

My takeaway from IROS

I came into today’s session thinking mainly about better robot foundation models.

I left thinking much more about robot architectures.

Scaling models will undoubtedly continue.

But physical intelligence has an unusual requirement: it must connect slow abstract reasoning with prediction, millisecond physical feedback, uncertainty, contact dynamics and irreversible actions.

That makes robotics fundamentally a systems problem.

And perhaps the most capable future robot won’t have one enormous brain issuing every motor command.

It may instead have something closer to:

a semantic brain that understands,

a world model that imagines,

a planner that searches,

specialized skills that act,

reflexes that react,

and a continual loop that learns from what actually happened.

In other words:

The next generation of robots may become intelligent not simply because they know what action to take, but because they can imagine what might happen before they take it.