Ask Abdo
A grounded AI portfolio guide that answers from current public data with real-time text streaming, duplex Gemini Multimodal Live voice audio, voice dictation, and renders every destination from current data supplied by the server.
Overview
AI Systems / AI guide
Ask Abdo is a public AI guide for abdok.dev that helps visitors explore projects, articles, photography, résumé details, and professional links through grounded text chat and real-time duplex Live voice conversations. It streams answers through Vercel AI SDK and Gemini 3.7 Flash, pairs low-latency WebSockets with the Gemini Multimodal Live API for natural two-way speech with explicit interruption controls, and accepts voice dictation with server-side container validation. Every request rebuilds trusted public evidence dynamically, rejects client-supplied tool output as factual authority, and materializes selected destinations from canonical server data. Authoritative concurrency leases (signed visitor cookies and PostgreSQL row-level locks via FOR UPDATE transactions) enforce active generation ceilings, persistent per-conversation transports, and cross-tab recovery.
Technical stack
4 stack groups
AI & Multimodal Orchestration
5 instruments in this layerKey features
9 capability groups
Duplex Multimodal Live Voice Experience
5 implementation notes- 01
Real-time bidirectional voice mode connected via low-latency WebSockets to the Gemini Multimodal Live API for natural spoken conversation.
- 02
Audio pipeline using Web Audio API and AudioWorklet for 16kHz PCM audio capture, base64 chunked streaming, and immediate playback buffer flush on explicit user interruption.
- 03
Dynamic Visualizer Capsule displaying real-time audio waveforms and listening/speaking/idle states, with a transient Live caption surface and inline tool-execution spinners.
- 04
Live presentation sidecars execute server tools (photo discovery, project details, article lookups, resume summary) and render rich destination cards without interrupting spoken voice output.
- 05
Session leasing, reconnect authorization tokens, session resumption persistence, and graceful storage cleanup for robust mobile and desktop network resilience.
Architecture notes
12 recorded decisions
- 01
Designed a trust boundary where client messages can guide conversation but never provide authoritative project, article, resume, social, or photo evidence.
- 02
Engineered a low-latency duplex WebSocket bridge to Gemini Multimodal Live API with 16kHz PCM audio streaming, explicit interruption controls, and live presentation sidecars.
- 03
Built an isolated voice dictation pipeline with container signature validation (OGG/WebM/MP4), rate limiting, and selection-aware composer insertion.
- 04
Supports optional terminal stable-key presentation in the same final model step as the streamed answer, avoiding a separate provider round trip for resource cards and suggestions.
- 05
Implemented a UI-stream transform that appends canonical resource manifests to the live message stream at completion without buffering or suppressing prose.
- 06
Added saved-chat rehydration that refreshes canonical attachment data instead of preserving stale local snapshots.
- 07
Built a model-only response suggestion pipeline with suggestions-only persistence, client-side cleanup, and no deterministic post-answer fallback generation.
- 08
Created a Batwick-driven assistant shell with the Dynamic Visualizer Capsule, a conversation-level Resources library, contextual response actions, canonical inline popovers, and route-aware starter prompts.
- 09
Added exact-ID/title photo retrieval, bounded named tag/location filters, filter-only provider avoidance, and public facet inventory while preserving the server-owned photo boundary.
- 10
Implemented immutable message editing with bounded edit history, descendant truncation confirmation, cross-tab tombstones, and generation invalidation.
- 11
Built a production-server real-model evaluation harness with parallel execution, per-prompt traces, provenance hashes, step limit checks, semantic advisories, and catalog-driven article coverage.
- 12
Added persistent per-conversation transports and authoritative backend concurrency leases (signed visitor cookie + PostgreSQL FOR UPDATE transactions) enforcing two active generations, cancellation grace caps, idempotency, status reconciliation, same-tab background generation, cross-tab coordination, and refresh-safe terminal recovery.
Project status
2 status records
Core Assistant & Concurrency Implemented
12 recorded items- 01
Correctness-first portfolio assistant route with trusted evidence building, request validation, rate limiting, bounded public tools, and live UI-message stream delivery.
- 02
Duplex Multimodal Live voice audio mode with low-latency WebSockets, 16kHz PCM capture, explicit interruption buffer flush, dynamic visualizer capsule, and live presentation sidecars.
- 03
Voice dictation and transcription route with audio container verification, rate limiting, and cursor-aware draft insertion.
- 04
Optional model-owned resource presentation plus canonical materialization for projects, published articles, photos, social links, and resume resources.
- 05
Responsive assistant UI with Batwick mascot, history drawer, message rendering, suggestions, and recruiter quick actions.
- 06
Model-only response follow-ups with suggestions-only turns, shared filtering and deduplication, and separate deterministic empty-chat starters.
- 07
Saved conversation references with fresh server-side attachment rehydration.
- 08
Conversation-level Resources with newest-first deduplication, category-aware response affordances, exact-route Currently viewing treatment, canonical inline actions, and direct photo-modal opening.
- 09
Exact photo ID/title retrieval, filter-first named tag/location search, filter-only provider avoidance, and explicit public facet inventory.
- 10
Immutable message editing with bounded history, confirmed descendant truncation, cross-tab tombstones, and stale-generation invalidation.
- 11
Cross-tab background generation registry with persistent conversation hosts, signed visitor cookies, PostgreSQL generation state leases (FOR UPDATE), revisioned status coordination, pending-generation recovery, terminal-result adoption, and History drawer active-generation controls.
- 12
Real-model starter evaluations with production builds, parallel runner support, observable tool traces, prompt-specific rubrics, and schema/provenance artifacts.