FetchIn progress
An on-device knowledge engine for Android. Pulls a page, turns it into clean Markdown, indexes it, and searches it back. Nothing leaves the phone.
I built an on-device knowledge engine for Android called Fetch.
It pulls a web page, strips it down to clean Markdown, indexes it, and searches it back. All of it on the phone.
Three tiers, cheapest first
Most of the cost in retrieving a page is the network. So the engine tries not to use it.
Tier one is the local index. If the page is already there, nothing touches the network at all.
Tier two is a plain HTTP fetch, with the HTML tree converted to Markdown.
Tier three is an offscreen WebView, for pages that render themselves with JavaScript and return almost nothing without it. Images, fonts and media are blocked, because none of that matters when you only want the text.
Getting the text out
Stripping a page down to its prose is harder than it sounds.
Ads and scripts come out by tag. That part is easy. What is not easy is that
modern frameworks split the actual text across JSON blobs. React puts it in
__NEXT_DATA__, Nuxt in __NUXT__, and the visible HTML is often a shell
waiting to be filled in.
So the extractor reconstructs prose out of those fragments as well. Without that, a lot of pages return a headline and nothing else.
val engine = FetchEngine.create(context, browser = WebViewBrowserBackend(context))
val document = engine.fetch("https://example.com/article")
println(document.markdown)
val results = engine.search(query = "quantum computing", limit = 5)Searching it back
The index is SQLite FTS5 with BM25 ranking. It lives on the device, in a normal database file.
That means search works with no connection, and the thing being searched is whatever you actually read, not whatever a search engine decided to keep.
The parts that are about not trusting the input
An engine that will fetch any URL it is handed is a proxy sitting inside
someone's app. So there is an SSRF guard that blocks private ranges,
127.0.0.1 and 10.0.0.0/8 included.
HTML recursion is depth-capped, because a hostile or broken page can nest forever and take the parser down with it.
Per-domain backoff respects Retry-After and remembers the window, so the
library is not hammering a host that already said no.
The local REST API binds to 127.0.0.1:8080 and its tokens are encrypted with
AES-256 through the Android KeyStore. A local API with no auth is still an API
any other app on the phone can call.
tl;dr
Fetch a page, keep the text, search it later, all on the phone.