The next step for conversational AI agents is multimodal

Mon, 26 Oct 2026 00:00:00 +0000

Last week, Konstantinos Siaterlis and I gave a talk at AWS Community Day Adria in Ljubljana, Slovenia, about the next step for intelligent conversational AI agents.

The slides are on the site, and you can click through them in the browser: Journey to Intelligent Conversational Agents.

The short version: the next step is multimodal AI agents. Text, visuals, voice and collaboration, while keeping memory, personalization and multiple contexts together.

We are working more with AI

We opened the talk by asking why AI has us working more, when it was supposed to save us time.

Making the work is cheap now, but checking it still takes your time. Every agent you add needs your context, your reviews and your decisions, and it keeps working while you sleep.

Glean’s Work AI Index (June 2026) has a word for this: botsitting, “feeding it context, supervising its output, debugging its mistakes, and cleaning up after it.” In their survey of 6,000 full-time workers, people reported 6.4 hours a week of it on average. Most of a working day, every week.

Voice is fast one way

Voice agents look like an answer. Speaking is fast. In a lab study, people entered text on an iPhone at 153 words a minute by voice and 52 by typing, almost three times faster.

Listening is slow. We read at about 238 words a minute, and an audiobook is spoken at 140 to 180. If ten agents read their status out to you, you wait. On a board, you see all ten at once.

So you speak to tell the agent what you want, and you look at a picture to understand what it did. That is what we mean by multimodal: you talk to the agent, and it shows you.

Ultron

To show it, we built one. Ultron is a multimodal AI voice agent that can:

listen to the whole room, recognize who is speaking, and address a person
design an architecture live, in front of the audience
draw the diagram
spot the risks in the architecture
advise the whole team
build, as well

We asked Ultron to draw its own architecture. It drew this with Claude Sonnet 5.5, in 13.6 seconds:

The parts: Speechmatics turns speech into text, pyannoteAI tells who spoke when, Claude Haiku, Sonnet and Opus 5.5 do the reasoning, and its voice is a Qwen3-TTS clone running on the Mac. Memory stays on the Mac too, in SQLite.

To decide which parts to build and which to rent, we used Wardley Mapping. We wrote our own code only where there was nothing we could rent.

The slides

All 50 slides are on the presentation page. They also go through where conversational agents started, from Turing to ChatGPT, and the 2018 re:Invent workshop where Slobodan Stojanović and I had Alexa ask Jarvis to create a serverless app.

Click the slides and use the arrow keys to move, or press F for full screen.

Multimodal on simalexan

The next step for conversational AI agents is multimodal

We are working more with AI

Voice is fast one way

Ultron

The slides