Document Processing

How Agency understands your documents.

Supported Formats

Agency processes a wide set of formats with native parsing, LibreOffice conversion, OCR, and archive/email explosion:

Category	Representative formats
----------	------------------------
PDF	`.pdf`
Office & ODF docs	`.doc`, `.docx`, `.docm`, `.dotx`, `.dotm`, `.docb`, `.odt`, `.ott`, `.sxw`, `.stw`, `.rtf`, `.txt`, `.md`, `.markdown`, `.org`, `.rst`
Spreadsheets & tabular	`.xls`, `.xlsx`, `.xlsm`, `.xltx`, `.xltm`, `.xlsb`, `.ods`, `.ots`, `.csv`, `.tsv`, `.dif`, `.slk`, `.dbf`, legacy Lotus/Works (`.wk1`, `.wks`, etc.)
Presentations & diagrams	`.ppt`, `.pptx`, `.pptm`, `.potx`, `.odp`, `.otp`, `.sxi`, `.sti`, `.vsd`, `.vsdx`, `.wmf`, `.emf`, `.cdr`
Text / markup / code	`.json`, `.yaml`, `.yml`, `.html`, `.xml`, `.js`, `.ts`, `.py`, `.java`, `.cpp`, `.c`, `.cs`, `.rb`, `.swift`, `.go`
Images (OCR)	`.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.bmp`, `.psd`, `.pcd`, `.gem`, `.img`, `.pic`, `.scr`, `.pnt`, `.mac`, `.drw`, `.dsf`, `.msp`, `.pcx`, `.psp`, `.tga`
Email	`.eml`, `.msg`
Email archives	`.pst`, `.mbox` (exploded into individual `.eml`/`.ics`/`.vcf`)
Calendars	`.ics`, `.ical`
EPUB	`.epub`
Archives	`.zip`, `.tar`, `.tar.gz`, `.tgz`, `.gtar.gz`, `.7z`, `.csv.gz`, `.seq.gz`
Event logs	Windows EVTX

Archives and email archives are exploded first; member files follow the same routing. Unknown text-like formats fall back to auto partitioning with conservative defaults.

Processing Pipeline

Format Detection — Identify file type and structure
Content Extraction — Extract text, tables, and metadata
OCR (if needed) — Process images and scanned documents
Entity Recognition — Identify people, places, dates, organizations
Embedding Generation — Create semantic vectors for search
Graph Building — Map relationships between entities

What Gets Indexed

Full text content
Document metadata (author, date, title)
Tables and structured data
Image descriptions (via vision AI)
Entity relationships
Temporal information

Status & Reliability

Each file shows ingestion status (processing, succeeded, failed with reason).
Corrupted or password-protected items are skipped and logged; healthy files continue.
OCR retries can be triggered for low-quality scans.
Indexes (text, vectors, graph) stay encrypted at rest in the local data stores.

Supported Formats

Agency processes a wide set of formats with native parsing, LibreOffice conversion, OCR, and archive/email explosion:

Category	Representative formats
----------	------------------------
PDF	`.pdf`
Office & ODF docs	`.doc`, `.docx`, `.docm`, `.dotx`, `.dotm`, `.docb`, `.odt`, `.ott`, `.sxw`, `.stw`, `.rtf`, `.txt`, `.md`, `.markdown`, `.org`, `.rst`
Spreadsheets & tabular	`.xls`, `.xlsx`, `.xlsm`, `.xltx`, `.xltm`, `.xlsb`, `.ods`, `.ots`, `.csv`, `.tsv`, `.dif`, `.slk`, `.dbf`, legacy Lotus/Works (`.wk1`, `.wks`, etc.)
Presentations & diagrams	`.ppt`, `.pptx`, `.pptm`, `.potx`, `.odp`, `.otp`, `.sxi`, `.sti`, `.vsd`, `.vsdx`, `.wmf`, `.emf`, `.cdr`
Text / markup / code	`.json`, `.yaml`, `.yml`, `.html`, `.xml`, `.js`, `.ts`, `.py`, `.java`, `.cpp`, `.c`, `.cs`, `.rb`, `.swift`, `.go`
Images (OCR)	`.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.bmp`, `.psd`, `.pcd`, `.gem`, `.img`, `.pic`, `.scr`, `.pnt`, `.mac`, `.drw`, `.dsf`, `.msp`, `.pcx`, `.psp`, `.tga`
Email	`.eml`, `.msg`
Email archives	`.pst`, `.mbox` (exploded into individual `.eml`/`.ics`/`.vcf`)
Calendars	`.ics`, `.ical`
EPUB	`.epub`
Archives	`.zip`, `.tar`, `.tar.gz`, `.tgz`, `.gtar.gz`, `.7z`, `.csv.gz`, `.seq.gz`
Event logs	Windows EVTX

Archives and email archives are exploded first; member files follow the same routing. Unknown text-like formats fall back to auto partitioning with conservative defaults.

Processing Pipeline

Format Detection — Identify file type and structure

Content Extraction — Extract text, tables, and metadata

OCR (if needed) — Process images and scanned documents

Entity Recognition — Identify people, places, dates, organizations

Embedding Generation — Create semantic vectors for search

Graph Building — Map relationships between entities