Speech to Speech MobileIn progress
A voice assistant that runs entirely on the phone. You speak, it hears, thinks and answers, with nothing leaving the device.
I am building a speech-to-speech engine that runs entirely on a phone.
You talk. It hears you, works out what you said, thinks about it, and answers out loud. No server anywhere in that loop.
This is the biggest thing I have worked on, and it is not finished.
What has to happen between you speaking and it answering
Four things, and each one is its own field of research.
Voice activity detection first, because running speech recognition on silence is a waste of a battery. Then recognition. Then a language model. Then speech synthesis.
On a server you would call four APIs. On a phone all four are files sitting in storage, loaded into memory, running on hardware that is also doing everything else.
Every stage is an interface
The thing I got right early was refusing to hard-code any of it.
val engine = S2SEngine(
context = this,
config = S2SConfig(models = ModelPaths(...)),
synthesizer = MyOwnSynthesizer(), // optional override
)
engine.initialize().getOrThrow() // slow, call off the main thread
engine.start()AudioInput, AudioOutput, VoiceActivityDetector, SpeechRecognizer,
LanguageModel, SpeechSynthesizer, TextChunker, ContextEngine and
TextNormalizer. Tool calling used to be one of them and has moved out into its
own repository, now archived. Swapping Kokoro for
Piper, or the streaming recogniser for an offline one, is a constructor
argument rather than a rewrite.
Talking over it, and pausing mid-thought
You can interrupt it. If you speak over the assistant for a few frames in a row, after a short grace window while the echo canceller settles, playback cuts off at once.
It also does not treat every pause as the end of your turn. Speech segments are collected across pauses, so "set a reminder, for tomorrow, to call the dentist" goes in as one request, not three.
That mattered more than I expected. The models in this space are replaced every few months, and anything that assumed a particular one would already be old.
What is actually running
VAD Silero VAD v5, ONNX
also TEN VAD, opt-in
STT streaming Zipformer transducer
also Zipformer2-CTC, Paraformer, NeMo-CTC
LLM llama.cpp, any GGUF
TTS Kokoro-82M at 24 kHz
also VITS/Piper, Matcha, Kitten, PocketNone of it is mine. sherpa-onnx handles VAD, recognition and synthesis
through ONNX Runtime. llama.cpp handles generation. Both are native, both are
fast, and neither needed rewriting.
The design is a Kotlin port of Hugging Face's speech-to-speech pipeline,
built on runtimes that already work rather than new ones.
Then it stopped being one thing
The engine kept growing. Context handling, tool calling, agent behaviour, different LLM backends. Each addition made the core bigger and harder to reason about, and most of it was not speech at all.
So it split into six repositories, each its own thing. Five of them are archived now, as the section after this one explains.
speech-to-speech-mobile is the engine. Audio in, audio out, and nothing else.
s2s-llm holds the language model
backends: llama.cpp, remote OpenAI-compatible endpoints, whatever comes next,
all behind the one LanguageModel contract.
s2s-context is memory. The transcript that persists, keyword retrieval over it, and a working context that stays bounded rather than growing until it will not fit.
s2s-tools is the tool registry and dispatcher, so the assistant can do things rather than only talk about them.
s2s-agent is the agent harness. Runtime, task store, tool coordination, skills, and a verifier to check the work was actually done.
s2s-host composes the rest and manages which plugins are installed.
They install as separate APKs and talk over AIDL. That means a plugin can crash without taking the engine with it, and you can ship one without shipping the other.
There is a boundary check in CI, because an architecture like this is only real if something enforces it. Left alone, the layers grow references to each other until the separation is a diagram rather than a fact.
The plugin system is archived
I have archived the five plugin repositories: s2s-llm, s2s-context,
s2s-tools, s2s-agent and s2s-host. Their last work was at the end of
August, and they stay up as read-only history.
The engine went back to being one thing. In October I cut it down to a pure speech-to-speech pipeline, with real-time speech streaming restored.
Things that only matter on a phone
Model loading is slow. initialize() is explicitly not for the main thread,
and the API says so rather than hoping.
Models are large, so the demo downloads them on first run rather than shipping them, and the downloader verifies SHA256, because a partially downloaded model fails in ways that look like a bug in the engine.
RECORD_AUDIO has to be granted before start(). That is an obvious thing
that is easy to get wrong at the wrong moment.
About the licence
The repository is GPL-3.0, and not by preference.
espeak-ng is in the dependency chain for text normalisation, and it is GPL.
Linking to it means the whole thing is GPL. I had it under Apache-2.0 before
noticing.
Worth saying plainly rather than leaving in a licence file: if you build on this, you inherit that.
From React Native
expo-s2s-mobile wraps the Android engine as an Expo native module, so a React Native app can use it without writing any Kotlin. It is not archived, and it is separate from the plugin system above.
You add it as a plugin in app.json, then call initializeAsync with the
paths to the VAD, speech recognition, GGUF language model and speech synthesis
models, call start(), and listen for events with addS2SListener.
await initializeAsync({
vad: { silenceTimeoutMs: 1500 },
stt: { modelDir: '/path/to/stt' },
llm: { modelPath: '/path/to/llm.gguf' },
tts: { modelPath: '/path/to/tts.onnx' },
})
start()It is Android only, and it needs a custom dev client or EAS Build, because it carries native code. The bridge has known limits on timeouts and on coalescing events, which are written down in the README. The example app has a model downloader and a settings screen.
It is not on npm yet, so you install it from the repository.
Where it is
Working end to end. You can speak to it and it answers, on the device, with nothing leaving it.
The engine is a pure speech-to-speech pipeline again, and the plugin system around it is archived.
tl;dr
Speak to your phone, have it answer, with nothing going to a server.