Talking to an Agent While It Works (S2S)In progress

S2S is the AI agent I am building to live on an Android phone. It keeps its own memory, uses tools, runs tasks, and you bring whichever AI model you want it to use.

The part I have been on lately is voice. You talk to it, it talks back, and it keeps working while you are talking.

I built it wrong first

My first version was a second, smaller agent that only handled the conversation and passed the real work to the main one. It knew less and could do less. Talking to it felt like talking to a receptionist.

It was not, so I removed it. Voice mode is now the main agent itself, in the same conversation, with the same memory, the same tools and the same name.

Voice only changes how your words go in and how its words come out.

It talks while it works

Before it uses a tool it says what it is about to do, something like "let me check that". The tool itself is never read out. On a long job it gives a short update, so you are not sitting in silence.

Reminders and finished background jobs get said out loud when they arrive. When the agent has a question for you, it asks it out loud, and you can answer by voice or by typing on the screen. Either one counts.

You can cut in

Talk over it and it stops at once. What happens next depends on what it was doing.

If it was in the middle of a job, "no, use the other folder" changes the work without starting over. If it was only talking, it drops that and answers the new thing, and it knows which part you heard before you cut in. That last bit matters more than it sounds, because otherwise it repeats what you already heard.

It sounds like someone listening

How it speaks comes from a file called VOICE.md in the agent's workspace, so you can change the style yourself. It talks in short phrases, reacts before it answers, and adjusts to how you sound. If you sound frustrated, it slows down and gets calmer.

The agent marks how a line should be said, and the voice follows it:

[concerned] Ah, sorry to hear that...

A tag like [warm] or [calm] changes the delivery, and [beat] is a short pause. Markdown, code and links never get read out.

It keeps going with the screen off

Voice mode runs like a phone call, so it survives going to the background and the screen turning off. The notification has Mute, where it stops hearing you but keeps talking, and End call. You can switch between speaker and earpiece.

A call ends when you say goodbye in your own words. A model judges that, not a list of words, because "tell Sam bye" is not a goodbye. If nothing happens for five minutes it asks "are you still there?" and hangs up if you do not answer. There is also a one hour limit.

How it got here

It came together fast, over about a week.

Where the voice runs

Listening and speaking happen on the phone, and the model you choose does the thinking. The voice models are downloaded the first time you use it, about 450 MB.

If your model can take audio, it also gets your recording. There is a setting to turn that off.

What went wrong

For a while the end of every answer went missing. It would say "It's Wednesday, October 14th, 20" and stop. The last piece of each reply was never sent. That is fixed.

The slow part is waiting for the model. Some providers answer all at once after 15 to 28 seconds, so it cannot start talking early. That is the provider and not my code. The fix I have planned is a fast helper model that says a short opener while the main one thinks. I have not built it.

Where it is

It works end to end with a real model. Most of the testing so far has been without real audio, using simulated speech. What only shows up with a real voice on a real phone is still being tuned: echo on the speaker, how fast it feels, and the battery.