We all misplace things, and note-taking apps need a discipline nobody keeps up. With Loci you photograph an item and the app tags it, indexes it and makes it searchable in plain words. No manual tagging, no folders.
Build a mobile app that detects objects in photos, generates embeddings for natural-language search, works offline and syncs when it reconnects, all without cloud services or recurring costs.
It had to feel native on Android while being built with web technologies, and two ML models had to run on the backend without expensive GPU infrastructure.
Take a photo of your keys on the kitchen counter. YOLOv8 detects the objects and CLIP generates embeddings, stored in a searchable index.
Then type "keys", or even "metal things in the kitchen". The query is encoded with CLIP and compared against the stored image embeddings by cosine similarity, so no exact keywords are needed.
Changes made offline are queued in IndexedDB and replayed in order on reconnect. Edge cases like delete-then-recreate and concurrent edits took the most time, but the app never feels broken without internet.
Both models run on the backend, not on the device. YOLOv8n and CLIP ViT-B/32 are small enough to run on a single CPU instance. I also replaced Supabase auth with a custom JWT and bcrypt flow to remove the external dependency and its cost.