Speech to Speech MobileIn progress

I am building a speech-to-speech engine that runs entirely on a phone.

You talk. It hears you, works out what you said, thinks about it, and answers out loud. No server anywhere in that loop.

This is the biggest thing I have worked on, and it is not finished.

What has to happen between you speaking and it answering

Four things, and each one is its own field of research.

Voice activity detection first, because running speech recognition on silence is a waste of a battery. Then recognition. Then a language model. Then speech synthesis.

On a server you would call four APIs. On a phone all four are files sitting in storage, loaded into memory, running on hardware that is also doing everything else.

Every stage is an interface

The thing I got right early was refusing to hard-code any of it.

val engine = S2SEngine(
    context = this,
    config = S2SConfig(models = ModelPaths(...)),
    synthesizer = MyOwnSynthesizer(),   // optional override
)
engine.initialize().getOrThrow()        // slow, call off the main thread
engine.start()

AudioInput, SpeechRecognizer, LanguageModel, SpeechSynthesizer, TextChunker, Tools. Six contracts. Swapping Kokoro for Piper, or the streaming recogniser for an offline one, is a constructor argument rather than a rewrite.

That mattered more than I expected. The models in this space are replaced every few months, and anything that assumed a particular one would already be old.

What is actually running

VAD    Silero VAD v5, ONNX
STT    streaming Zipformer transducer
       also Zipformer2-CTC, Paraformer, NeMo-CTC
LLM    llama.cpp, any GGUF
TTS    Kokoro-82M at 24 kHz
       also VITS/Piper, Matcha, Kitten, Pocket

None of it is mine. sherpa-onnx handles VAD, recognition and synthesis through ONNX Runtime. llama.cpp handles generation. Both are native, both are fast, and neither needed rewriting.

The design is a Kotlin port of Hugging Face's speech-to-speech pipeline, built on runtimes that already work rather than new ones.

Then it stopped being one thing

The engine kept growing. Context handling, tool calling, agent behaviour, different LLM backends. Each addition made the core bigger and harder to reason about, and most of it was not speech at all.

So it split into six repositories, each its own thing.

speech-to-speech-mobile is the engine. Audio in, audio out, and nothing else.

s2s-llm holds the language model backends: llama.cpp, remote OpenAI-compatible endpoints, whatever comes next, all behind the one LanguageModel contract.

s2s-context is memory. The transcript that persists, keyword retrieval over it, and a working context that stays bounded rather than growing until it will not fit.

s2s-tools is the tool registry and dispatcher, so the assistant can do things rather than only talk about them.

s2s-agent is the agent harness. Runtime, task store, tool coordination, skills, and a verifier to check the work was actually done.

s2s-host composes the rest and manages which plugins are installed.

They install as separate APKs and talk over AIDL. That means a plugin can crash without taking the engine with it, and you can ship one without shipping the other.

There is a boundary check in CI, because an architecture like this is only real if something enforces it. Left alone, the layers grow references to each other until the separation is a diagram rather than a fact.

Things that only matter on a phone

Model loading is slow. initialize() is explicitly not for the main thread, and the API says so rather than hoping.

Models are large, so the demo downloads them on first run rather than shipping them, and the downloader verifies SHA256, because a partially downloaded model fails in ways that look like a bug in the engine.

RECORD_AUDIO has to be granted before start(). That is an obvious thing that is easy to get wrong at the wrong moment.

About the licence

The repository is GPL-3.0, and not by preference.

espeak-ng is in the dependency chain for text normalisation, and it is GPL. Linking to it means the whole thing is GPL. I had it under Apache-2.0 before noticing.

Worth saying plainly rather than leaving in a licence file: if you build on this, you inherit that.

Where it is

Working end to end. You can speak to it and it answers, on the device, with nothing leaving it.

The plugin ecosystem is newer and moving. The agent runtime is the least settled part.

tl;dr

Speak to your phone, have it answer, with nothing going to a server.

Engine · LLM · Context · Tools · Agent · Host