Prototype
ESP32 voice assistant
A small e-ink device you press, talk to, and that answers back. It keeps the hardware simple and does the heavy lifting on a self-hosted server.
How it works
🔘 press & hold→
🎙️ record→
🔐 secure WebSocket→
📝 speech-to-text→
🧠 LLM with tools→
🔊 voice reply
- Device: Waveshare ESP32-S3 board with an e-ink screen, a microphone and a speaker, powered by a small LiPo battery.
- Firmware: ESP-IDF with Arduino as a component. A button wakes it from deep sleep, a real-time clock refreshes the idle screen, and it supports over-the-air updates.
- Server: a FastAPI service that runs speech-to-text, an LLM with tools (for example, writing notes into a knowledge base) and text-to-speech, streamed back to the device.
- Other clients: an iPhone companion app talks to the same server, and tools can be enabled per client.
Where it stands
The full loop works: talk, transcription, answer and voice reply, on the device and from the iPhone app. The current challenge is power: the device can reset mid-recording on battery, most likely a brownout when the microphone and Wi-Fi draw current at the same time. On-device crash tracing is in place to confirm it.
Next
- Confirm and fix the battery brownout.
- Finish the e-ink and audio drivers and the deep-sleep state machine.
- Wire up an Apple Watch push-to-talk client.