We built a modular voice-driven AI agent stack that starts with audio input, uses Groq Whisper for speech-to-text, and passes the transcript to an LLM-based intent classifier, all guarded by a safe Tool Execution Layer and presented through a Streamlit UI that shows transcription, detected intent, actions, and outputs.

The tech stack relies on Python with Streamlit, Whisper for STT, LLMs via Ollama or API, and local OS/file handling libraries to keep operations self-contained.

Architecturally, five components drive the system: a frontend with mic input and file upload; a FastAPI backend; an stt module using Groq Whisper Large v3 with a local fallback; an intent module based on LLaMA; and a sandboxed tools module that executes actions inside an output/ folder.

A key challenge is context resolution for follow-ups, ensuring subsequent prompts like generating a quiz pull from prior content rather than re-reading the prompt text.

The project highlights practical tradeoffs between local and cloud inference, emphasizing graceful degradation and robust error handling as core learnings for a resilient voice-driven AI agent.

Early issues with validation of an OpenAI Structured Output Schema led to a redesigned schema with explicit fields, alongside migrations to Tailwind CSS v4.

Improving intent classification and securely handling API keys, including a .env file and GitHub ignore rules, were essential challenges.

Solutions emphasized local model operation with optional API fallbacks, safer file operations restricted to output/, robust audio error handling, and structured prompts for better intent detection.

Readers are directed to the full source code and documentation in the Mem0-Assignment repository for implementation details.

Streamlit session state was introduced to isolate per-session memories, enabling multiple concurrent users without cross-session interference.

Future work includes adding Function Calling for web browsing and local databases to extend the agent’s capabilities beyond listening and acting.

Graceful degradation is built in to handle API/key issues, timeouts, and unsupported formats, with fallback to general chat when intents are unknown.