agent/docs/architecture/retrieval_inventory.md

Retrieval Inventory

Scope and method

This document describes the retrieval and indexing pipeline as implemented in code today. The inventory is based primarily on:

• app/modules/rag/services/rag_service.py
• app/modules/rag/persistence/*.py
• app/modules/rag/indexing/code/**/*.py
• app/modules/rag/indexing/docs/**/*.py
• app/modules/rag_session/module.py
• app/modules/agent/engine/graphs/project_qa_step_graphs.py
• app/modules/agent/engine/orchestrator/*.py

ASSUMPTION: the intended layer semantics are the ones implied by code and tests, not by future architecture plans. This matters because only C0 through C3 are materially implemented today; C4+ exist only as enum constants.

Current retrieval pipeline

1. Retrieval entrypoint is POST /internal/rag/retrieve in app/modules/rag_session/module.py.
2. The endpoint calls RagService.retrieve(rag_session_id, query).
3. RagQueryRouter chooses docs or code mode from the raw query text.
4. RagService computes a single embedding for the full query via GigaChatEmbedder.
5. RagQueryRepository.retrieve(...) runs one SQL query against rag_chunks in PostgreSQL with pgvector.
6. Ranking order is:
   • lexical rank
   • test-file penalty
   • layer rank
   • vector distance embedding <=> query_embedding
7. Response items are normalized to {source, content, layer, title, metadata, score}.
8. If embeddings fail, retrieval falls back to latest chunks from the same layers.
9. If code retrieval returns nothing, service falls back to docs layers.

Storage and indices

• Primary store: PostgreSQL from DATABASE_URL, configured in app/modules/shared/db.py.
• Vector extension: CREATE EXTENSION IF NOT EXISTS vector in app/modules/rag/persistence/schema_repository.py.
• Primary table: rag_chunks.
• Cache tables:
  • rag_blob_cache
  • rag_chunk_cache
  • rag_session_chunk_map
• SQL indexes currently created:
  • (rag_session_id)
  • (rag_session_id, layer)
  • (rag_session_id, layer, path)
  • (qname)
  • (symbol_id)
  • (module_id)
  • (doc_kind)
  • (entrypoint_type, framework)

ASSUMPTION: there is no explicit ANN index for the vector column in schema code. The code creates general SQL indexes, but no ivfflat/hnsw index is defined here.

Layer: C0_SOURCE_CHUNKS

Implementation

• Produced by CodeIndexingPipeline.index_file(...) in app/modules/rag/indexing/code/pipeline.py.
• Chunking logic: CodeTextChunker.chunk(...) in app/modules/rag/indexing/code/code_text/chunker.py.
• Document builder: CodeTextDocumentBuilder.build(...) in app/modules/rag/indexing/code/code_text/document_builder.py.
• Persisted via RagDocumentRepository.insert_documents(...) into rag_chunks.

Input contract

This is an indexing layer, not a direct public retriever. The observed upstream indexing input is a file dict with at least:

• required:
  • path: str
  • content: str
• optional:
  • commit_sha: str | None
  • content_hash: str
  • metadata fields copied through by RagService._document_metadata(...)

For retrieval, the layer is queried only indirectly through:

• rag_session_id: str
• query: str
• inferred mode/layers from RagQueryRouter
• fixed limit=8

Output contract

Stored document shape:

• top-level:
  • layer = "C0_SOURCE_CHUNKS"
  • lang = "python"
  • source.repo_id
  • source.commit_sha
  • source.path
  • title
  • text
  • span.start_line
  • span.end_line
  • embedding
• metadata:
  • chunk_index
  • chunk_type: symbol_block or window
  • module_or_unit
  • artifact_type = "CODE"
  • plus file-level metadata injected by RagService

Returned retrieval item shape:

• source
• content
• layer
• title
• metadata
• score

No line_start / line_end are returned to the caller directly; they remain in DB columns span_start / span_end and are only used in logs.

Defaults & limits

• AST chunking prefers one chunk per top-level class/function/async function.
• Fallback window chunking:
  • size = 80 lines
  • overlap = 15 lines
• Global retrieval limit from RagService.retrieve(...): 8
• Embedding batch size from env:
  • RAG_EMBED_BATCH_SIZE
  • default 16

Known issues

• Nested methods/functions are not emitted as C0 chunks unless represented inside a selected top-level block.
• Returned API payload omits line spans even though storage has them.
• No direct filter by path, namespace, symbol, or top_k is exposed through the current endpoint.

Layer: C1_SYMBOL_CATALOG

Implementation

• Symbol extraction: SymbolExtractor.extract(...) in app/modules/rag/indexing/code/symbols/extractor.py.
• AST parsing: PythonAstParser.parse_module(...).
• Document builder: SymbolDocumentBuilder.build(...).
• Retrieval reads rows from rag_chunks; there is no dedicated symbol table.

Input contract

Indexing input is the same per-file payload as C0.

Observed symbol extraction source:

• Python AST only
• supported symbol kinds:
  • class
  • function
  • method
  • const for top-level imports/import aliases

Retrieval input is still the generic text query endpoint. Query terms are enriched by extract_query_terms(...):

• extracts identifier-like tokens from query text
• normalizes camelCase/PascalCase to snake_case
• adds special intent terms for management/control-related queries
• max observed query terms: 6

Output contract

Stored document shape:

• top-level:
  • layer = "C1_SYMBOL_CATALOG"
  • title = qname
  • text = "<kind> <qname>\n<signature>\n<docstring?>"
  • span.start_line
  • span.end_line
• metadata:
  • symbol_id
  • qname
  • kind
  • signature
  • decorators_or_annotations
  • docstring_or_javadoc
  • parent_symbol_id
  • package_or_module
  • is_entry_candidate
  • lang_payload
  • artifact_type = "CODE"

Observed lang_payload variants:

• class:
  • bases
• function/method:
  • async
• import alias:
  • imported_from
  • import_alias

Defaults & limits

• Only Python source files are indexed into C-layers.
• Import and import-from declarations are materialized as const symbols only at module top level.
• Retrieval ranking gives C1 priority rank 1, after C3 and before C2/C0.

Known issues

• No explicit visibility/public-private model.
• parent_symbol_id currently stores the parent qname string from the stack, not the parent symbol hash. This is an observed implementation detail.
• Cross-file symbol resolution is not implemented; dst_symbol_id in edges resolves only against symbols extracted from the same file.

Layer: C2_DEPENDENCY_GRAPH

Implementation

• Edge extraction: EdgeExtractor.extract(...) in app/modules/rag/indexing/code/edges/extractor.py.
• Document builder: EdgeDocumentBuilder.build(...).
• Built during CodeIndexingPipeline.index_file(...).

Input contract

Indexing input is the same per-file source payload as C0/C1.

Graph construction method:

• static analysis only
• Python AST walk only
• no runtime tracing
• no tree-sitter

Observed edge types:

• calls
• imports
• inherits

Output contract

Stored document shape:

• top-level:
  • layer = "C2_DEPENDENCY_GRAPH"
  • title = "<src_qname>:<edge_type>"
  • text = "<src_qname> <edge_type> <dst>"
  • span.start_line
  • span.end_line
  • links contains one evidence link of type EDGE
• metadata:
  • edge_id
  • edge_type
  • src_symbol_id
  • src_qname
  • dst_symbol_id
  • dst_ref
  • resolution: resolved or partial
  • lang_payload
  • artifact_type = "CODE"

Observed lang_payload usage:

• for calls: may include callsite_kind = "function_call"

Defaults & limits

• Edge extraction is per-file only.
• imports edges are emitted only while visiting a class/function scope; top-level imports do not become C2 edges.
• Layer rank in retrieval SQL: 2

Known issues

• There is no traversal API, graph repository, or query language over C2. Retrieval only treats edges as text/vector rows in rag_chunks.
• Destination resolution is local to the file-level qname map.
• Top-level module import relationships are incompletely represented because visit_Import / visit_ImportFrom skip when there is no current scope.

Layer: C3_ENTRYPOINTS

Implementation

• Detection registry: EntrypointDetectorRegistry.detect_all(...).
• Detectors:
  • FastApiEntrypointDetector
  • FlaskEntrypointDetector
  • TyperClickEntrypointDetector
• Document builder: EntrypointDocumentBuilder.build(...).

Input contract

Indexing input is the same per-file source payload as other C-layers.

Detected entrypoint families today:

• HTTP:
  • FastAPI decorators such as .get, .post, .put, .patch, .delete, .route
  • Flask .route
• CLI:
  • Typer/Click .command
  • Typer/Click .callback

Not detected:

• Django routes
• Celery tasks
• RQ jobs
• cron jobs / scheduler entries

Output contract

Stored document shape:

• top-level:
  • layer = "C3_ENTRYPOINTS"
  • title = route_or_command
  • text = "<framework> <entry_type> <route_or_command>"
  • span.start_line
  • span.end_line
  • links contains one evidence link of type CODE_SPAN
• metadata:
  • entry_id
  • entry_type: observed http or cli
  • framework: observed fastapi, flask, typer, click
  • route_or_command
  • handler_symbol_id
  • lang_payload
  • artifact_type = "CODE"

FastAPI-specific observed payload:

• lang_payload.methods = [HTTP_METHOD] for .get/.post/...

Defaults & limits

• Retrieval layer rank: 0 highest among code layers.
• Entrypoint mapping is handler-symbol centric:
  • decorator match -> symbol -> handler_symbol_id
  • physical location comes from symbol span

Known issues

• Route parsing is string-based from decorator text, not semantic AST argument parsing.
• No dedicated entrypoint tags beyond entry_type, framework, and raw decorator-derived payload.
• Background jobs and non-decorator entrypoints are not indexed.

Dependency graph / trace current state

Exists or stub?

• C2 exists and is populated.
• It is not a stub.
• It is also not a full-project dependency graph service; it is a set of per-edge documents stored in rag_chunks.

How the graph is built

• static Python AST analysis
• no runtime instrumentation
• no import graph resolver across modules
• no tree-sitter

Edge types in data

• calls
• imports
• inherits

Traversal API

• No traversal API was found in app/modules/rag/* or app/modules/agent/*.
• No method accepts graph traversal parameters such as depth, start node, edge filters, or BFS/DFS strategy.
• Current access path is only retrieval over indexed edge documents.

Entrypoints current state

Implemented extraction

• HTTP routes:
  • FastAPI
  • Flask
• CLI:
  • Typer
  • Click

Mapping model

• entrypoint -> handler_symbol_id -> symbol span/path
• The entrypoint record itself stores:
  • framework
  • entry type
  • raw route/command string
  • handler symbol id

Tags/types

• entry_type is the main normalized tag.
• Observed values: http, cli.
• framework is the second discriminator.
• There are no richer endpoint taxonomies such as job, worker, webhook, scheduler.

Defaults and operational limits

• Query mode default: docs
• Code mode is enabled by keyword heuristics in RagQueryRouter
• Retrieval hard limit: 8
• Fallback limit: 8
• Query term extraction limit: 6
• Ranked source bundle for project QA:
  • top 12 RAG items
  • top 10 file candidates
• No exposed namespace, path_prefixes, top_k, max_chars, max_chunks, max_depth in the public/internal retrieval endpoint

ASSUMPTION: the absence of these controls in endpoint and service signatures means they are not part of the current supported contract, even though RagQueryRepository.retrieve(...) has an internal path_prefixes parameter.

Known cross-cutting issues

• Retrieval contract is effectively text-only at API level; structured retrieval exists only as internal SQL parameters.
• Response payload drops explicit line spans even though spans are stored.
• Vector retrieval is coupled to a single provider-specific embedder.
• Docs mode is the default, so code retrieval depends on heuristic query phrasing unless the project/qa graph prepends по коду.
• There is no separate retrieval contract per layer exposed over API; all layer selection is implicit.

Where to plug ExplainPack pipeline

Option 1: replace or extend project_qa/context_analysis

• Code location:
  • app/modules/agent/engine/graphs/project_qa_step_graphs.py
• Why:
  • retrieval is already complete at this step
  • input bundle already contains ranked rag_items and file_candidates
  • output is already a structured analysis_brief
• Risk:
  • low
  • minimal invasion if ExplainPack consumes source_bundle and emits the same analysis_brief shape

Option 2: insert a new orchestrator step between context_retrieval and context_analysis

• Code location:
  • app/modules/agent/engine/orchestrator/template_registry.py
  • app/modules/agent/engine/orchestrator/step_registry.py
• Why:
  • preserves current retrieval behavior
  • makes ExplainPack an explicit pipeline stage with its own artifact
  • cleanest for observability and future A/B migration
• Risk:
  • low to medium
  • requires one new artifact contract and one extra orchestration step, but no change to retrieval storage

Option 3: introduce ExplainPack inside ExplainActions.extract_logic

• Code location:
  • app/modules/agent/engine/orchestrator/actions/explain_actions.py
• Why:
  • useful if ExplainPack is meant only for explain-style scenarios
  • keeps general project QA untouched
• Risk:
  • medium
  • narrower integration point; may create duplicate reasoning logic separate from project QA analysis path

Bottom line

• C0-C3 are implemented and persisted in one physical store: rag_chunks.
• Retrieval is a hybrid SQL ranking over lexical heuristics plus pgvector distance.
• C2 exists, but only as retrievable edge documents, not as a traversable graph subsystem.
• C3 covers FastAPI/Flask/Typer/Click only.
• The least invasive ExplainPack integration point is after retrieval and before answer composition, preferably as a new explicit orchestrator artifact or as a replacement for context_analysis.