Ingest · retrieve

Knowledge assistant

Three deployables: a document ingest pipeline into vector search, an orchestrator API, and a chat client. Answers are generated from indexed chunks, not from a naked prompt.

PROCESSORS, INDEX FIELDS, APIS, SECURITY

01Processor factory

class DocumentIngestorFactory:
    _ingestors = {
        "pdf": PDFIngestor,
        "docx": DOCXIngestor,
        "excel": ExcelIngestor,
    }

    @classmethod
    def get_ingestor(cls, file_type: str) -> BaseDocumentIngestor:
        if file_type not in cls._ingestors:
            raise ValueError(f"Unsupported file type: {file_type}")
        return cls._ingestors[file_type]()

A new format is a class with ingest(), then one registry line.

Processor	Path
PDF	Document intelligence → paragraphs, tables, figures → page groups → embeddings
DOCX	Convert to PDF, then the PDF path, so layout quality stays one pipeline
Excel	Per sheet, header detection, tables chunked ~20 rows, markdown for the embedder

02Chunking

Recursive splitter: 1500 characters, 200 overlap, paragraph then line then word then character separators. Captions on figures are first-class fields so retrieval can hit an image by its label.

03Index fields

Field	Role
`chunk_id`	Key
`content`	Searchable text
`content_vector`	3072-d, HNSW cosine
`blob_name` / `blob_url`	Provenance + delete filter
`page_number` / `chunk_index`	Cite location
`content_type`	paragraph / table / figure
`caption`	Figure/table title
`file_type`	pdf / docx / xlsx
`upload_timestamp`	Recency

HNSW: m=4, efConstruction=400, efSearch=500. Semantic ranker uses blob_name as title and content + caption as body.

Typical: 30–50 documents per source file, ~2KB per chunk with vector.

04HTTP surface

Auth: X-API-KEY. Missing or wrong → 401.

Route	Result
`POST /upload`	`filename`, `file_type`, `num_chunks`, `num_tables`, `num_images`, `processing_time_seconds`
`DELETE /files/{name}`	`chunks_deleted`
`GET /files/{name}/chunks`	`ChunkInfo[]` for debug / evals

Errors:

{
  "status": "error",
  "message": "Unsupported file type",
  "error_code": "UNSUPPORTED_TYPE",
  "details": { "file_type": "ppt" }
}

Rate limit is not in the first ship; 10 uploads / minute per address is the intended ceiling.

05Embeddings

Batch 64. One batch call is ~80% faster than per-text calls on a 50-chunk file. Small sleep between batches to stay inside provider quotas.

06Functions topology

HTTP routes share the FastAPI app with routePrefix: "". Beside them:

blob-created trigger → ingest
Event Grid delete → drop chunks
HTTP cleanup for operators
timer for orphan sweep

Local FastAPI (run_server.py) is for iteration. Production is the Functions host so triggers exist.

07Orchestrator LLD (adjacent)

The bot posts to /orchestrator with the same API-key pattern. Session and traces live in a document store. The orchestrator URL is configuration, not a hardcoded host. Ingest and orchestrator must not share a default port in local dev — one of them moves.

Synthetic data platform

SRS generator