Ingest · retrieve
Knowledge assistant
Three deployables: a document ingest pipeline into vector search, an orchestrator API, and a chat client. Answers are generated from indexed chunks, not from a naked prompt.
LAYERS, DATA FLOW, EVENT TRIGGERS
01Principles
- Serverless-first compute on Functions.
- Event-driven for new blobs and deletions — not a cron that pretends to watch storage.
- Stateless workers. Session lives in Cosmos (or equivalent), not in the process.
- Modular processors. PDF, DOCX, Excel share an interface; the factory picks by type.
02Layers
Rendering diagram…
03Upload flow
POST /uploadwith multipart file andX-API-KEY.- Detect type; reject unknown.
- Write raw bytes to blob storage.
- Run the processor: layout → text/tables/figures → chunks (1500 / 200 overlap, recursive separators).
- Embed in batches of 64 (3072-d).
- Upsert search documents. Return chunk / table / image counts and elapsed seconds.
04Delete flow
Blob deletion is not “forget the file name.” Event Grid (or an explicit delete API) removes all chunks whose blob_name matches. Index and blob stay twins.
05Orchestrator vs ingest
The chat client calls the orchestrator, not the ingest app. Ingest is a write path. Orchestrator is retrieve → generate → cite, plus a traces dashboard. Hosting both as Functions is an ops choice; the boundary is the API.
06Trust
| Layer | Control |
|---|---|
| Transport | TLS 1.2+ |
| App | API key on mutating and query routes |
| Cloud | Managed identity to storage, search, embeddings |
| Data | Encryption at rest; SAS URLs expire in hours, not days |
| Blobs | Least-privilege containers; no public read |