Skip to main content

Retrieval and Answering

Retrieval combines semantic vector search with lexical keyword search, fuses both rankings with RRF, reranks candidates, and passes cited chunks to synthesis. Semantic-only retrieval is explicitly out because government legal and administrative documents depend on exact statute numbers, named entities, and official terms.

Authoritative API contract

This page explains retrieval and answer-generation architecture. For exact /search, /query, and /query/stream request fields, response bodies, citation shape, status/error behavior, SSE event order, curl commands, and TypeScript examples, use the Query and Search API reference.

Retrieval flow​

Citation model​

Each retrieved chunk carries citation metadata: document identity, document title, chunk identity and ordinal, page number, section path, legal reference, and character span. This architecture page names the provenance model only; the complete citation object and conditional access-control fields are defined in the Query and Search API reference.

Faithfulness gate​

The Synthesizer checks whether each generated claim is supported by cited chunks. Unsupported claims are dropped or the system returns a fallback answer that says it could not verify a faithful answer from retrieved sources.

The support check is figure-aware, not word-overlap only. An exact normalized numeric, currency, percent, or ratio value shared between a claim and a cited chunk grounds terse and cross-language figure answers — an English answer over an Indonesian chunk — that a word-overlap bar drops, because the figure is often the only token shared across languages. Values are normalized across Indonesian and English digit-grouping conventions before comparison, and a number introduced by a structural keyword (Pasal, Nomor, Tahun, ayat, Lampiran) is treated as a clause/statute reference rather than a groundable quantity, so a bare 2 never grounds on Pasal 2. Ratios ground only in the orientation written, and negation or polarity conflicts stay authoritative over any figure match. See src/aceh_rag/services.py _claim_supported_by_chunk/_figure_values.

Tool selection and configured sources​

Retrieval runs through a tool the Planner selects for the query's intent, with hybrid Vector Search as the default and fallback. Legal Check, Summarizer, Cross-Reference, and Excel Query use indexed or uploaded document data. The read-only sql_query tool registers only when a structured SQL source is enabled, so a quantity-shaped question over prose PPID content is answered from the retrieved chunk rather than routed to an unconfigured SQL source and refused. See src/aceh_rag/orchestration.py.

Live OpenData, SatuData, and BPS reads​

When enabled with an egress allow-list, the Open Data tool reads a published figure directly from query-specific CKAN OpenData or SatuData resources and grounds it with dataset identity and read-time metadata. The BPS Statistics tool reads the authoritative BPS WebAPI for indicators such as population, poverty, inflation, or GDP; it shares the open_data_lookup intent and can fall back to the Open Data portal when configured. These tools are disabled by default and must degrade to an honest unavailable/not-enabled note rather than invent a figure. See src/aceh_rag/open_data.py and src/aceh_rag/bps.py.

For BPS series, period discovery reads all advertised period pages within the configured read budget and explicitly reports a partial N of TOTAL list if the budget is exhausted. One table call accepts one to three discovered period IDs, labels every returned row with its period, and keeps separate period citations.

Policy Guard​

The Government Policy Guard wraps generated answers before they reach users. It filters anti-SARA, jailbreak attempts, and government-communication-policy violations. It cannot be bypassed per query.

Query and search surfaces​

POST /search exposes ranked retrieval chunks for diagnostics and result previews. POST /query runs retrieval plus synthesis and returns a cited answer, while POST /query/stream streams the same answer lifecycle over SSE. This section is an architecture summary; the Query and Search API reference is authoritative for payload fields, status codes, standard errors, SSE terminal behavior, and frontend examples.