Фикс состояния
This commit is contained in:
108
tests/pipeline_setup/suite_02_pipeline/README.md
Normal file
108
tests/pipeline_setup/suite_02_pipeline/README.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Suite 02 Pipeline
|
||||
|
||||
Интеграционный набор тестов для режимов:
|
||||
|
||||
- `router_only`
|
||||
- `router_rag`
|
||||
- `full_chain`
|
||||
|
||||
## Что входит в suite
|
||||
|
||||
- `pipeline_intent_rag/` — тесты, fixtures и runtime пайплайна
|
||||
- `cli/` — запускные скрипты для индексации и ручного прогона пайплайна
|
||||
|
||||
## Запуск тестов через pytest
|
||||
|
||||
Из корня проекта:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. pytest tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/test_intent_router_only_matrix.py -q
|
||||
```
|
||||
|
||||
Для retrieval-режима:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. RUN_INTENT_PIPELINE_ROUTER_RAG=1 pytest -m router_rag tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/ -q
|
||||
```
|
||||
|
||||
Для полной цепочки:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. RUN_INTENT_PIPELINE_FULL_CHAIN=1 pytest -m full_chain tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/ -q
|
||||
```
|
||||
|
||||
## Запуск через CLI
|
||||
|
||||
### 1. Индексация репозитория
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.index_repo --repo-path /abs/path/to/repo [--project-id ID]
|
||||
```
|
||||
|
||||
Параметры:
|
||||
|
||||
- `--repo-path` — путь к индексируемому репозиторию
|
||||
- `--project-id` — `project_id` для новой RAG-сессии
|
||||
|
||||
### 2. Router only
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_only [--case-id ID ...] [--verbose] [--test-name PREFIX]
|
||||
```
|
||||
|
||||
Параметры:
|
||||
|
||||
- `--case-id` — запуск только выбранных кейсов, можно передавать несколько раз
|
||||
- `--verbose` — печатать полную диагностику
|
||||
- `--test-name` — префикс имени артефакта
|
||||
|
||||
### 3. Router + RAG
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_rag --rag-session-id <uuid> [--case-id ID ...] [--verbose] [--test-name PREFIX]
|
||||
```
|
||||
|
||||
или с переиндексацией:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_rag --reindex-repo-path /abs/path/to/repo [--reindex-project-id ID] [--case-id ID ...] [--verbose]
|
||||
```
|
||||
|
||||
Параметры:
|
||||
|
||||
- `--rag-session-id` — готовый `rag_session_id` для всех кейсов
|
||||
- `--reindex-repo-path` — путь к репозиторию, который нужно проиндексировать перед прогоном
|
||||
- `--reindex-project-id` — `project_id` для новой RAG-сессии
|
||||
- `--case-id` — запуск только выбранных кейсов
|
||||
- `--verbose` — печатать расширенную диагностику
|
||||
- `--test-name` — префикс имени артефакта
|
||||
|
||||
### 4. Full chain
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.run_full_chain --rag-session-id <uuid> [--case-id ID ...] [--verbose] [--test-name PREFIX]
|
||||
```
|
||||
|
||||
или с переиндексацией:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=. python -m tests.pipeline_setup.suite_02_pipeline.cli.run_full_chain --reindex-repo-path /abs/path/to/repo [--reindex-project-id ID] [--case-id ID ...] [--verbose]
|
||||
```
|
||||
|
||||
Параметры те же, что и у `run_router_rag`.
|
||||
|
||||
## Переменные окружения
|
||||
|
||||
- `RUN_INTENT_PIPELINE_ROUTER_RAG=1` — включает тесты и прогоны `router_rag`
|
||||
- `RUN_INTENT_PIPELINE_FULL_CHAIN=1` — включает тесты и прогоны `full_chain`
|
||||
- `DATABASE_URL` — обязателен для retrieval и full chain
|
||||
- `INTENT_PIPELINE_FORCE_RAG_SESSION_ID` — принудительный `rag_session_id` для всех кейсов
|
||||
- `INTENT_PIPELINE_RAG_SESSION_ID` — дефолтный `rag_session_id`
|
||||
- `INTENT_PIPELINE_REINDEX_REPO_PATH` — путь к репозиторию для автоиндексации
|
||||
- `INTENT_PIPELINE_REINDEX_PROJECT_ID` — `project_id` для автоиндексации
|
||||
|
||||
Артефакты прогонов пишутся в:
|
||||
[test_results](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup/test_results)
|
||||
|
||||
Детали по внутреннему устройству пайплайна:
|
||||
[pipeline_intent_rag/README.md](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/README.md)
|
||||
1
tests/pipeline_setup/suite_02_pipeline/__init__.py
Normal file
1
tests/pipeline_setup/suite_02_pipeline/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Pipeline integration suites for router_only, router_rag, and full_chain."""
|
||||
1
tests/pipeline_setup/suite_02_pipeline/cli/__init__.py
Normal file
1
tests/pipeline_setup/suite_02_pipeline/cli/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""CLI entrypoints for pipeline test runs and repository reindexing."""
|
||||
36
tests/pipeline_setup/suite_02_pipeline/cli/index_repo.py
Normal file
36
tests/pipeline_setup/suite_02_pipeline/cli/index_repo.py
Normal file
@@ -0,0 +1,36 @@
|
||||
"""Скрипт индексации репозитория: создаёт RAG-сессию и индексирует указанную директорию.
|
||||
|
||||
Используется для подготовки репозитория перед запуском тестовых пайплайнов с retrieval или full_chain.
|
||||
Запуск из корня проекта (agent):
|
||||
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.index_repo --repo-path /path/to/repo [--project-id ID]
|
||||
|
||||
Параметры:
|
||||
--repo-path (обязательный) Путь к корню индексируемого репозитория.
|
||||
--project-id (опционально) Идентификатор проекта для созданной rag_session; по умолчанию — имя директории репо.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Обеспечиваем импорт из корня проекта
|
||||
_agent_root = Path(__file__).resolve().parents[4]
|
||||
if str(_agent_root) not in sys.path:
|
||||
sys.path.insert(0, str(_agent_root))
|
||||
|
||||
# Добавляем src в путь для app
|
||||
_src = _agent_root / "src"
|
||||
if _src.exists() and str(_src) not in sys.path:
|
||||
sys.path.insert(0, str(_src))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
argv = ["reindex", *sys.argv[1:]]
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import main as cli_main
|
||||
return cli_main(argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
46
tests/pipeline_setup/suite_02_pipeline/cli/run_full_chain.py
Normal file
46
tests/pipeline_setup/suite_02_pipeline/cli/run_full_chain.py
Normal file
@@ -0,0 +1,46 @@
|
||||
"""Запуск тестового пайплайна «полная цепочка»: intent router + retrieval + LLM.
|
||||
|
||||
Цепочка: intent_router_v2 → RAG retrieval → ответ GigaChat (или другой LLM). Нужны БД и индексированный репозиторий.
|
||||
|
||||
Запуск из корня проекта (agent):
|
||||
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.run_full_chain --rag-session-id <uuid> [--case-id ID ...] [--verbose]
|
||||
|
||||
Или с индексацией перед прогоном:
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.run_full_chain --reindex-repo-path /path/to/repo [--reindex-project-id ID]
|
||||
|
||||
Параметры:
|
||||
--rag-session-id (рекомендуется) UUID RAG-сессии (после индексации через index_repo или run с --reindex-repo-path).
|
||||
--reindex-repo-path Индексировать указанный репозиторий перед прогоном и использовать новую сессию.
|
||||
--reindex-project-id project_id для новой сессии при использовании --reindex-repo-path.
|
||||
--case-id (повторяемый) Запустить только указанные кейсы.
|
||||
--verbose Выводить диагностику по каждому кейсу.
|
||||
--test-name Префикс имени файла с результатами (по умолчанию cli_full_chain).
|
||||
|
||||
Переменные окружения:
|
||||
RUN_INTENT_PIPELINE_FULL_CHAIN=1 — включить маркер full_chain (иначе тесты с этим маркером пропускаются).
|
||||
DATABASE_URL — подключение к БД (обязательно для retrieval).
|
||||
(и переменные для доступа к LLM, см. .env и pipeline_intent_rag/.env.test)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_agent_root = Path(__file__).resolve().parents[4]
|
||||
if str(_agent_root) not in sys.path:
|
||||
sys.path.insert(0, str(_agent_root))
|
||||
_src = _agent_root / "src"
|
||||
if _src.exists() and str(_src) not in sys.path:
|
||||
sys.path.insert(0, str(_src))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
argv = ["run", "--mode", "full_chain", *sys.argv[1:]]
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import main as cli_main
|
||||
return cli_main(argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Запуск тестового пайплайна «только intent router» (без RAG и LLM).
|
||||
|
||||
Цепочка: intent_router_v2 — классификация запроса, без retrieval и ответа LLM.
|
||||
|
||||
Запуск из корня проекта (agent):
|
||||
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_only [--case-id ID ...] [--verbose]
|
||||
|
||||
Параметры:
|
||||
--case-id (повторяемый) Запустить только указанные кейсы по id. Без указания — все кейсы с тегом router_only.
|
||||
--verbose Выводить диагностику по каждому кейсу.
|
||||
--test-name Префикс имени файла с результатами (по умолчанию cli_router_only).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_agent_root = Path(__file__).resolve().parents[4]
|
||||
if str(_agent_root) not in sys.path:
|
||||
sys.path.insert(0, str(_agent_root))
|
||||
_src = _agent_root / "src"
|
||||
if _src.exists() and str(_src) not in sys.path:
|
||||
sys.path.insert(0, str(_src))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
argv = ["run", "--mode", "router_only", *sys.argv[1:]]
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import main as cli_main
|
||||
return cli_main(argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
45
tests/pipeline_setup/suite_02_pipeline/cli/run_router_rag.py
Normal file
45
tests/pipeline_setup/suite_02_pipeline/cli/run_router_rag.py
Normal file
@@ -0,0 +1,45 @@
|
||||
"""Запуск тестового пайплайна «intent router + retrieval» (без LLM).
|
||||
|
||||
Цепочка: intent_router_v2 → RAG retrieval. Нужна предварительная индексация репозитория.
|
||||
|
||||
Запуск из корня проекта (agent):
|
||||
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_rag --rag-session-id <uuid> [--case-id ID ...] [--verbose]
|
||||
|
||||
Или с индексацией перед прогоном:
|
||||
python -m tests.pipeline_setup.suite_02_pipeline.cli.run_router_rag --reindex-repo-path /path/to/repo [--reindex-project-id ID]
|
||||
|
||||
Параметры:
|
||||
--rag-session-id (рекомендуется) UUID RAG-сессии (после индексации через index_repo или run с --reindex-repo-path).
|
||||
--reindex-repo-path Индексировать указанный репозиторий перед прогоном и использовать новую сессию.
|
||||
--reindex-project-id project_id для новой сессии при использовании --reindex-repo-path.
|
||||
--case-id (повторяемый) Запустить только указанные кейсы.
|
||||
--verbose Выводить диагностику по каждому кейсу.
|
||||
--test-name Префикс имени файла с результатами (по умолчанию cli_router_rag).
|
||||
|
||||
Переменные окружения:
|
||||
RUN_INTENT_PIPELINE_ROUTER_RAG=1 — включить маркер router_rag (иначе тесты с этим маркером пропускаются).
|
||||
DATABASE_URL — подключение к БД (обязательно для retrieval).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_agent_root = Path(__file__).resolve().parents[4]
|
||||
if str(_agent_root) not in sys.path:
|
||||
sys.path.insert(0, str(_agent_root))
|
||||
_src = _agent_root / "src"
|
||||
if _src.exists() and str(_src) not in sys.path:
|
||||
sys.path.insert(0, str(_src))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
argv = ["run", "--mode", "router_rag", *sys.argv[1:]]
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import main as cli_main
|
||||
return cli_main(argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,12 @@
|
||||
# Test-only environment for pipeline_intent_rag suite
|
||||
# Override this file locally when using another database.
|
||||
DATABASE_URL=postgresql+psycopg://agent:agent@localhost:5432/agent
|
||||
GIGACHAT_SSL_VERIFY=false
|
||||
|
||||
# Optional defaults for integration runs:
|
||||
# RUN_INTENT_PIPELINE_ROUTER_RAG=1
|
||||
# RUN_INTENT_PIPELINE_FULL_CHAIN=1
|
||||
# INTENT_PIPELINE_RAG_SESSION_ID=<existing_rag_session_id>
|
||||
# INTENT_PIPELINE_FORCE_RAG_SESSION_ID=<force_for_all_cases>
|
||||
# INTENT_PIPELINE_REINDEX_REPO_PATH=/absolute/path/to/repo
|
||||
# INTENT_PIPELINE_REINDEX_PROJECT_ID=my-project
|
||||
@@ -0,0 +1,12 @@
|
||||
# Test-only environment for pipeline_intent_rag suite
|
||||
# Override this file locally when using another database.
|
||||
DATABASE_URL=postgresql+psycopg://agent:agent@db:5432/agent
|
||||
GIGACHAT_SSL_VERIFY=false
|
||||
|
||||
# Optional defaults for integration runs:
|
||||
# RUN_INTENT_PIPELINE_ROUTER_RAG=1
|
||||
# RUN_INTENT_PIPELINE_FULL_CHAIN=1
|
||||
# INTENT_PIPELINE_RAG_SESSION_ID=<existing_rag_session_id>
|
||||
# INTENT_PIPELINE_FORCE_RAG_SESSION_ID=<force_for_all_cases>
|
||||
# INTENT_PIPELINE_REINDEX_REPO_PATH=/absolute/path/to/repo
|
||||
# INTENT_PIPELINE_REINDEX_PROJECT_ID=my-project
|
||||
@@ -0,0 +1,82 @@
|
||||
# Intent Pipeline Test Module
|
||||
|
||||
## Что тестируется
|
||||
|
||||
- `router_only`: только `intent_router_v2`.
|
||||
- `router_rag`: `intent_router_v2 -> RAG retrieval`.
|
||||
- `full_chain`: `intent_router_v2 -> RAG -> GigaChat`.
|
||||
|
||||
## Каталог фраз
|
||||
|
||||
Фразы лежат в [fixtures/phrases.yaml](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/fixtures/phrases.yaml).
|
||||
|
||||
Поля кейса:
|
||||
|
||||
- `id` — уникальный ID.
|
||||
- `text` — запрос пользователя.
|
||||
- `tags` — в каких режимах кейс участвует (`router_only`, `router_rag`, `full_chain`).
|
||||
- `expected_intent` — опциональная проверка intent.
|
||||
- `expect_non_empty_rag` — ожидание непустого RAG-контекста.
|
||||
- `rag_session_id` — опционально; можно не указывать, если задан через env.
|
||||
|
||||
## Env для тестов
|
||||
|
||||
Сначала загружается `[.env.test](/Users/alex/Dev_projects_v2/ai driven app process/v2/agent/tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/.env.test)`, затем workspace `.env`.
|
||||
|
||||
Ключи:
|
||||
|
||||
- `DATABASE_URL` — подключение к тестовой БД.
|
||||
- `RUN_INTENT_PIPELINE_ROUTER_RAG=1` — включить `router_rag`.
|
||||
- `RUN_INTENT_PIPELINE_FULL_CHAIN=1` — включить `full_chain`.
|
||||
- `INTENT_PIPELINE_FORCE_RAG_SESSION_ID` — принудительный `rag_session_id` для всех кейсов.
|
||||
- `INTENT_PIPELINE_RAG_SESSION_ID` — дефолтный `rag_session_id` для кейсов.
|
||||
- `INTENT_PIPELINE_REINDEX_REPO_PATH` — путь к репозиторию для новой индексации.
|
||||
- `INTENT_PIPELINE_REINDEX_PROJECT_ID` — `project_id` для новой сессии (опционально).
|
||||
|
||||
Если указан `INTENT_PIPELINE_REINDEX_REPO_PATH`, раннер создаёт новую `rag_session`, индексирует репозиторий и использует этот `rag_session_id` для retrieval.
|
||||
|
||||
## Артефакты
|
||||
|
||||
Каждый прогон пишет JSONL в `tests/pipeline_setup/test_results`.
|
||||
Формат имени:
|
||||
|
||||
- `<test_name>_<YYYYMMDD_HHMMSS>.jsonl`
|
||||
|
||||
Примеры:
|
||||
|
||||
- `test_intent_router_only_matrix_20260305_183010.jsonl`
|
||||
- `test_intent_router_rag_pipeline_20260305_183045.jsonl`
|
||||
|
||||
## Запуск
|
||||
|
||||
- `pytest tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/test_intent_router_only_matrix.py -q`
|
||||
- `RUN_INTENT_PIPELINE_ROUTER_RAG=1 pytest -m router_rag tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/test_intent_router_rag_pipeline.py -q`
|
||||
- `RUN_INTENT_PIPELINE_FULL_CHAIN=1 pytest -m full_chain tests/pipeline_setup/suite_02_pipeline/pipeline_intent_rag/test_intent_router_rag_llm_pipeline.py -q`
|
||||
|
||||
## CLI
|
||||
|
||||
Запуск без `pytest`:
|
||||
|
||||
- `python3 -m tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag run --mode router_only`
|
||||
- `python3 -m tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag run --mode router_rag --rag-session-id <sid>`
|
||||
- `python3 -m tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag run --mode full_chain --rag-session-id <sid>`
|
||||
|
||||
Переиндексация репозитория в новую `rag_session`:
|
||||
|
||||
- `python3 -m tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag reindex --repo-path /abs/path/to/repo`
|
||||
- опционально: `--project-id my-project`
|
||||
|
||||
Команда печатает итог:
|
||||
|
||||
- `rag_session_id=<uuid>`
|
||||
|
||||
Далее этот `rag_session_id` можно передать в тестовый прогон:
|
||||
|
||||
- `python3 -m tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag run --mode router_rag --rag-session-id <uuid>`
|
||||
|
||||
Полезные опции:
|
||||
|
||||
- `--case-id <id>` (можно повторять)
|
||||
- `--reindex-repo-path /abs/path/to/repo`
|
||||
- `--reindex-project-id <project_id>`
|
||||
- `--test-name <artifact_prefix>`
|
||||
@@ -0,0 +1 @@
|
||||
# Isolated integration pipeline tests for intent-router -> RAG -> LLM roll-out.
|
||||
@@ -0,0 +1,5 @@
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,241 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.env_bootstrap import PipelineEnvLoader
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase, PipelineResult
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.runtime import PipelineRuntime
|
||||
|
||||
_VALID_MODES = ("router_only", "router_rag", "full_chain")
|
||||
|
||||
|
||||
@dataclass(slots=True, frozen=True)
|
||||
class RunCliArgs:
|
||||
mode: str
|
||||
case_ids: tuple[str, ...]
|
||||
rag_session_id: str | None
|
||||
reindex_repo_path: str | None
|
||||
reindex_project_id: str | None
|
||||
test_name: str
|
||||
verbose: bool
|
||||
|
||||
|
||||
@dataclass(slots=True, frozen=True)
|
||||
class ReindexCliArgs:
|
||||
repo_path: str
|
||||
project_id: str | None
|
||||
|
||||
|
||||
@dataclass(slots=True, frozen=True)
|
||||
class ParsedCliArgs:
|
||||
command: Literal["run", "reindex"]
|
||||
payload: RunCliArgs | ReindexCliArgs
|
||||
|
||||
|
||||
class PipelineCliRunner:
|
||||
def __init__(self, args: RunCliArgs, test_root: Path) -> None:
|
||||
self._args = args
|
||||
self._test_root = test_root
|
||||
|
||||
def run(self) -> int:
|
||||
self._apply_overrides()
|
||||
runtime = PipelineRuntime(mode=self._args.mode, test_name=self._args.test_name, test_root=self._test_root)
|
||||
cases = self._filter_cases(runtime.load_cases())
|
||||
if not cases:
|
||||
print("No cases selected")
|
||||
return 2
|
||||
|
||||
failed = 0
|
||||
executed = 0
|
||||
for case in cases:
|
||||
executed += 1
|
||||
try:
|
||||
result = self._run_case(runtime, case)
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
print(f"[FAIL] {case.case_id}: {exc}")
|
||||
continue
|
||||
try:
|
||||
self._validate_case(result)
|
||||
self._print_result(result, status="OK", reason=None, force_diagnostics=self._args.verbose)
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
self._print_result(result, status="FAIL", reason=str(exc), force_diagnostics=True)
|
||||
|
||||
print("-")
|
||||
print(f"Mode: {self._args.mode}")
|
||||
print(f"Executed: {executed}")
|
||||
print(f"Failed: {failed}")
|
||||
print(f"Artifact: {runtime.artifact_path}")
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
def _apply_overrides(self) -> None:
|
||||
if self._args.rag_session_id:
|
||||
os.environ["INTENT_PIPELINE_FORCE_RAG_SESSION_ID"] = self._args.rag_session_id
|
||||
if self._args.reindex_repo_path:
|
||||
os.environ["INTENT_PIPELINE_REINDEX_REPO_PATH"] = self._args.reindex_repo_path
|
||||
if self._args.reindex_project_id:
|
||||
os.environ["INTENT_PIPELINE_REINDEX_PROJECT_ID"] = self._args.reindex_project_id
|
||||
|
||||
def _filter_cases(self, cases: list[PhraseCase]) -> list[PhraseCase]:
|
||||
if not self._args.case_ids:
|
||||
return cases
|
||||
wanted = set(self._args.case_ids)
|
||||
return [case for case in cases if case.case_id in wanted]
|
||||
|
||||
def _run_case(self, runtime: PipelineRuntime, case: PhraseCase) -> PipelineResult:
|
||||
if self._args.mode == "router_only":
|
||||
return runtime.run_router_only(case)
|
||||
if self._args.mode == "router_rag":
|
||||
return runtime.run_router_rag(case)
|
||||
return runtime.run_full_chain(case)
|
||||
|
||||
def _validate_case(self, result: PipelineResult) -> None:
|
||||
case = result.case
|
||||
if case.expected_intent and result.intent != case.expected_intent:
|
||||
raise AssertionError(f"expected_intent={case.expected_intent} actual={result.intent}")
|
||||
if self._args.mode in {"router_rag", "full_chain"} and case.expect_non_empty_rag and not result.rag_rows:
|
||||
raise AssertionError(f"RAG returned empty list for rag_session_id={result.rag_session_id}")
|
||||
if self._args.mode == "full_chain" and not (result.llm_answer or "").strip():
|
||||
raise AssertionError("LLM answer must not be empty")
|
||||
|
||||
def _print_result(
|
||||
self,
|
||||
result: PipelineResult,
|
||||
*,
|
||||
status: str,
|
||||
reason: str | None,
|
||||
force_diagnostics: bool,
|
||||
) -> None:
|
||||
record = result.to_record()
|
||||
if status != "OK":
|
||||
print(f"status: {status}")
|
||||
print(f"case_id: {record['case_id']}")
|
||||
print(f"expected_intent: {record['expected_intent']}")
|
||||
print(f"actual_intent: {record['actual_intent']}")
|
||||
print("summary:")
|
||||
self._print_block(record.get("summary"), indent=2)
|
||||
if force_diagnostics:
|
||||
if reason:
|
||||
print(f"failure_reason: {reason}")
|
||||
print("diagnostics:")
|
||||
self._print_block(record.get("diagnostics"), indent=2)
|
||||
print("-")
|
||||
|
||||
def _print_block(self, value, *, indent: int) -> None:
|
||||
prefix = " " * indent
|
||||
if isinstance(value, dict):
|
||||
for key, item in value.items():
|
||||
if isinstance(item, (dict, list)):
|
||||
print(f"{prefix}{key}:")
|
||||
self._print_block(item, indent=indent + 2)
|
||||
else:
|
||||
print(f"{prefix}{key}: {item}")
|
||||
return
|
||||
if isinstance(value, list):
|
||||
for item in value:
|
||||
if isinstance(item, dict):
|
||||
print(f"{prefix}-")
|
||||
self._print_block(item, indent=indent + 2)
|
||||
else:
|
||||
print(f"{prefix}- {item}")
|
||||
return
|
||||
print(f"{prefix}{value}")
|
||||
|
||||
|
||||
class ReindexCliRunner:
|
||||
def __init__(self, args: ReindexCliArgs, test_root: Path) -> None:
|
||||
self._args = args
|
||||
self._test_root = test_root
|
||||
|
||||
def run(self) -> int:
|
||||
try:
|
||||
PipelineEnvLoader(self._test_root).load()
|
||||
repo_path = Path(self._args.repo_path).expanduser().resolve()
|
||||
if not repo_path.is_dir():
|
||||
print(f"[FAIL] repository path is not a directory: {repo_path}")
|
||||
return 2
|
||||
from app.modules.rag.persistence.repository import RagRepository
|
||||
from tests.pipeline_setup.utils.rag_indexer import RagSessionIndexer
|
||||
|
||||
RagSessionIndexer(RagRepository()).index_repo(repo_path, project_id=self._args.project_id)
|
||||
return 0
|
||||
except Exception as exc:
|
||||
print(f"[FAIL] reindex error: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Run intent-router pipeline CLI")
|
||||
sub = parser.add_subparsers(dest="command")
|
||||
|
||||
run_cmd = sub.add_parser("run", help="Run test pipeline")
|
||||
run_cmd.add_argument("--mode", choices=_VALID_MODES, required=True, help="Execution mode")
|
||||
run_cmd.add_argument("--case-id", action="append", default=[], help="Case id to run (repeatable)")
|
||||
run_cmd.add_argument("--rag-session-id", default=None, help="Force rag_session_id for all cases")
|
||||
run_cmd.add_argument("--reindex-repo-path", default=None, help="Path to local repository for reindex before retrieval")
|
||||
run_cmd.add_argument("--reindex-project-id", default=None, help="project_id for newly indexed rag_session")
|
||||
run_cmd.add_argument("--test-name", default=None, help="Artifact prefix (defaults to cli_<mode>)")
|
||||
run_cmd.add_argument(
|
||||
"--verbose",
|
||||
"--debug",
|
||||
dest="verbose",
|
||||
action="store_true",
|
||||
help="Print diagnostics for every case",
|
||||
)
|
||||
|
||||
reindex_cmd = sub.add_parser("reindex", help="Create a new RAG session by indexing local repository")
|
||||
reindex_cmd.add_argument("--repo-path", required=True, help="Path to local repository directory")
|
||||
reindex_cmd.add_argument("--project-id", default=None, help="Project id for created rag_session")
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def _parse_args(argv: list[str] | None = None) -> ParsedCliArgs:
|
||||
parser = _build_parser()
|
||||
raw = list(sys.argv[1:] if argv is None else argv)
|
||||
if raw and raw[0] not in {"run", "reindex"} and raw[0] not in {"-h", "--help"}:
|
||||
raw = ["run", *raw]
|
||||
ns = parser.parse_args(raw)
|
||||
command = str(getattr(ns, "command", "") or "")
|
||||
if command == "run":
|
||||
mode = str(ns.mode)
|
||||
test_name = (ns.test_name or f"cli_{mode}").strip() or f"cli_{mode}"
|
||||
payload = RunCliArgs(
|
||||
mode=mode,
|
||||
case_ids=tuple(str(item).strip() for item in ns.case_id if str(item).strip()),
|
||||
rag_session_id=str(ns.rag_session_id).strip() or None if ns.rag_session_id is not None else None,
|
||||
reindex_repo_path=str(ns.reindex_repo_path).strip() or None if ns.reindex_repo_path is not None else None,
|
||||
reindex_project_id=str(ns.reindex_project_id).strip() or None if ns.reindex_project_id is not None else None,
|
||||
test_name=test_name,
|
||||
verbose=bool(getattr(ns, "verbose", False)),
|
||||
)
|
||||
return ParsedCliArgs(command="run", payload=payload)
|
||||
if command == "reindex":
|
||||
payload = ReindexCliArgs(
|
||||
repo_path=str(ns.repo_path).strip(),
|
||||
project_id=str(ns.project_id).strip() or None if ns.project_id is not None else None,
|
||||
)
|
||||
return ParsedCliArgs(command="reindex", payload=payload)
|
||||
parser.error("command is required: run or reindex")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
try:
|
||||
args = _parse_args(argv)
|
||||
test_root = Path(__file__).resolve().parent
|
||||
if args.command == "run":
|
||||
return PipelineCliRunner(args=args.payload, test_root=test_root).run()
|
||||
return ReindexCliRunner(args=args.payload, test_root=test_root).run()
|
||||
except KeyboardInterrupt:
|
||||
print("Interrupted")
|
||||
return 130
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,70 @@
|
||||
phrases:
|
||||
- id: code-open-context-file
|
||||
text: "Открой файл src/mail_order_bot/context.py"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: code-explain-context-class
|
||||
text: "Объясни как работает класс Context"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: code-explain-excel-parser
|
||||
text: "Объясни класс ExcelFileParcer"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: code-find-tests-for-context
|
||||
text: "Где тесты для Context?"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: code-exclude-tests-context
|
||||
text: "Не про тесты, а про прод код Context"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: code-open-abstract-task
|
||||
text: "Покажи файл src/mail_order_bot/task_processor/abstract_task.py"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag"]
|
||||
|
||||
- id: code-explain-handle-errors
|
||||
text: "Теперь объясни функцию handle_errors"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "CODE_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: docs-about-readme-deploy
|
||||
text: "Что сказано в README_DEPLOY.md?"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "DOCS_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: docs-generic-question
|
||||
text: "Что сказано в документации по деплою?"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "DOCS_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag", "full_chain"]
|
||||
|
||||
- id: docs-open-readme-ref
|
||||
text: "Что про это в docs и README?"
|
||||
rag_session_id: "849fc2c9-e17c-4034-b3b5-2e13d535bb94"
|
||||
expected_intent: "DOCS_QA"
|
||||
expect_non_empty_rag: true
|
||||
tags: ["router_only", "router_rag"]
|
||||
@@ -0,0 +1 @@
|
||||
__all__: list[str] = []
|
||||
@@ -0,0 +1,121 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PipelineResult
|
||||
|
||||
|
||||
@dataclass(slots=True, frozen=True)
|
||||
class AnswerPolicyDecision:
|
||||
answer: str
|
||||
answer_mode: str
|
||||
failure_reason: str = ""
|
||||
|
||||
|
||||
class FullChainAnswerPolicy:
|
||||
def decide(self, result: PipelineResult) -> AnswerPolicyDecision | None:
|
||||
sub_intent = _sub_intent(result)
|
||||
if sub_intent == "OPEN_FILE":
|
||||
return self._missing_file(result)
|
||||
if sub_intent == "EXPLAIN":
|
||||
return self._missing_symbol(result)
|
||||
if sub_intent == "FIND_ENTRYPOINTS":
|
||||
return self._confirmed_entrypoints(result)
|
||||
return None
|
||||
|
||||
def _missing_file(self, result: PipelineResult) -> AnswerPolicyDecision | None:
|
||||
requested_path = _requested_path(result)
|
||||
if not requested_path or _has_exact_file_match(result, requested_path):
|
||||
return None
|
||||
return AnswerPolicyDecision(
|
||||
answer=f"Файл {requested_path} не найден.",
|
||||
answer_mode="degraded",
|
||||
failure_reason="file_not_found",
|
||||
)
|
||||
|
||||
def _missing_symbol(self, result: PipelineResult) -> AnswerPolicyDecision | None:
|
||||
symbol_resolution = dict(result.symbol_resolution or {})
|
||||
if str(symbol_resolution.get("status") or "").strip() != "not_found":
|
||||
return None
|
||||
symbol = _requested_symbol(result)
|
||||
if not symbol:
|
||||
return None
|
||||
answer = f"Сущность {symbol} не найдена в доступном коде."
|
||||
alternatives = [str(item).strip() for item in symbol_resolution.get("alternatives") or [] if str(item).strip()]
|
||||
if alternatives:
|
||||
answer = f"{answer} Близкие совпадения: {', '.join(alternatives[:3])}."
|
||||
return AnswerPolicyDecision(
|
||||
answer=answer,
|
||||
answer_mode="degraded",
|
||||
failure_reason="symbol_not_found",
|
||||
)
|
||||
|
||||
def _confirmed_entrypoints(self, result: PipelineResult) -> AnswerPolicyDecision | None:
|
||||
routes = _confirmed_routes(result)
|
||||
if not routes:
|
||||
return None
|
||||
lines = [_render_route(route) for route in routes]
|
||||
answer = "\n".join(f"- {line}" for line in lines)
|
||||
if len(lines) == 1:
|
||||
answer = lines[0]
|
||||
return AnswerPolicyDecision(answer=answer, answer_mode="answered")
|
||||
|
||||
|
||||
def _sub_intent(result: PipelineResult) -> str:
|
||||
router_plan = dict(result.diagnostics.get("router_plan") or {})
|
||||
return str(router_plan.get("sub_intent") or "").strip().upper()
|
||||
|
||||
|
||||
def _requested_path(result: PipelineResult) -> str:
|
||||
router_plan = dict(result.diagnostics.get("router_plan") or {})
|
||||
path_scope = tuple(str(item).strip() for item in router_plan.get("path_scope") or [] if str(item).strip())
|
||||
return path_scope[0] if path_scope else ""
|
||||
|
||||
|
||||
def _has_exact_file_match(result: PipelineResult, requested_path: str) -> bool:
|
||||
requested = requested_path.strip().lower()
|
||||
for row in result.rag_rows:
|
||||
row_path = str(row.get("path") or "").strip().lower()
|
||||
if row_path == requested:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _requested_symbol(result: PipelineResult) -> str:
|
||||
router_plan = dict(result.diagnostics.get("router_plan") or {})
|
||||
symbols = tuple(str(item).strip() for item in router_plan.get("symbol_candidates") or [] if str(item).strip())
|
||||
return symbols[0] if symbols else ""
|
||||
|
||||
|
||||
def _confirmed_routes(result: PipelineResult) -> list[dict]:
|
||||
seen: set[tuple[str, str]] = set()
|
||||
items: list[dict] = []
|
||||
for row in result.rag_rows:
|
||||
if str(row.get("layer") or "") != "C3_ENTRYPOINTS":
|
||||
continue
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
method = str(metadata.get("http_method") or "").strip()
|
||||
route_path = str(metadata.get("route_path") or "").strip()
|
||||
if not method or not route_path:
|
||||
continue
|
||||
key = (method, route_path)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
items.append(metadata)
|
||||
return items
|
||||
|
||||
|
||||
def _render_route(metadata: dict) -> str:
|
||||
method = str(metadata.get("http_method") or "").strip()
|
||||
route_path = str(metadata.get("route_path") or "").strip()
|
||||
declaring = str(metadata.get("declaring_symbol") or "").strip()
|
||||
handler = str(metadata.get("handler_symbol") or "").strip()
|
||||
label = f"{method} {route_path}"
|
||||
parts = [label]
|
||||
if declaring:
|
||||
parts.append(f"объявлен в {declaring}")
|
||||
if handler:
|
||||
handler_name = handler.rsplit(".", 1)[-1]
|
||||
parts.append(f"обрабатывается {handler_name}")
|
||||
return " — ".join((parts[0], ", ".join(parts[1:]))) if len(parts) > 1 else parts[0]
|
||||
@@ -0,0 +1,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
|
||||
class AnswerQualityController:
|
||||
_REPLACEMENTS = (
|
||||
("различные аргументы", "аргументы"),
|
||||
("ряд аргументов", "аргументы"),
|
||||
("различных подпакетов", "подпакетов"),
|
||||
("основные службы", "службы"),
|
||||
("играет роль", "работает как"),
|
||||
("представляет собой", "это"),
|
||||
)
|
||||
_DROP_PATTERNS = (
|
||||
r"(?i)ответ основан исключительно[^.]*\.",
|
||||
r"(?i)нет явных неподтвержденных кандидатов\.?",
|
||||
r"(?i)кандидаты[^.\n]*нет[^.\n]*\.?",
|
||||
r"(?i)related tests[^.\n]*none[^.\n]*\.?",
|
||||
r"(?i)indirect evidence[^.\n]*none[^.\n]*\.?",
|
||||
)
|
||||
_NOT_FOUND_PATTERNS = (
|
||||
"не найден",
|
||||
"не найдена",
|
||||
"не найдены",
|
||||
"не обнаружен",
|
||||
"не обнаружена",
|
||||
"не обнаружены",
|
||||
)
|
||||
|
||||
def refine(self, text: str, *, sub_intent: str | None = None) -> str:
|
||||
value = self._normalize_markdown(text)
|
||||
value = self._drop_empty_sections(value)
|
||||
value = self._replace_vague_phrases(value)
|
||||
if self._should_trim_not_found(value, sub_intent=sub_intent):
|
||||
value = self._trim_after_not_found(value)
|
||||
return self._normalize_spacing(value)
|
||||
|
||||
def _normalize_markdown(self, text: str) -> str:
|
||||
value = str(text or "").strip()
|
||||
value = re.sub(r"(?m)^\s{0,3}#{1,6}\s*", "", value)
|
||||
return value.replace("**", "")
|
||||
|
||||
def _drop_empty_sections(self, text: str) -> str:
|
||||
value = text
|
||||
for pattern in self._DROP_PATTERNS:
|
||||
value = re.sub(pattern, "", value)
|
||||
return value
|
||||
|
||||
def _replace_vague_phrases(self, text: str) -> str:
|
||||
value = text
|
||||
for source, target in self._REPLACEMENTS:
|
||||
value = re.sub(re.escape(source), target, value, flags=re.IGNORECASE)
|
||||
return value
|
||||
|
||||
def _should_trim_not_found(self, text: str, *, sub_intent: str | None) -> bool:
|
||||
current = str(sub_intent or "").strip().upper()
|
||||
if current not in {"EXPLAIN", "OPEN_FILE", "FIND_TESTS", "FIND_ENTRYPOINTS"}:
|
||||
return False
|
||||
lowered = text.lower()
|
||||
return any(pattern in lowered for pattern in self._NOT_FOUND_PATTERNS)
|
||||
|
||||
def _trim_after_not_found(self, text: str) -> str:
|
||||
match = re.search(r"(?<=[.!?])\s+", text.strip())
|
||||
if not match:
|
||||
return text.strip()
|
||||
return text[: match.start()].strip()
|
||||
|
||||
def _normalize_spacing(self, text: str) -> str:
|
||||
value = re.sub(r"\n{3,}", "\n\n", text)
|
||||
return re.sub(r"[ \t]{2,}", " ", value).strip()
|
||||
@@ -0,0 +1,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ArtifactWriter:
|
||||
def __init__(self, artifacts_dir: Path, test_name: str, run_started_at: datetime) -> None:
|
||||
self._artifacts_dir = artifacts_dir
|
||||
self._test_name = test_name
|
||||
self._run_started_at = run_started_at
|
||||
|
||||
@property
|
||||
def path(self) -> Path:
|
||||
stamp = self._run_started_at.strftime("%Y%m%d_%H%M%S")
|
||||
name = f"{self._test_name}_{stamp}.jsonl"
|
||||
return self._artifacts_dir / name
|
||||
|
||||
def write_record(self, payload: dict) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with self.path.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(payload, ensure_ascii=False) + "\n")
|
||||
@@ -0,0 +1,247 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from copy import deepcopy
|
||||
from typing import Any
|
||||
|
||||
from app.modules.rag.retrieval.test_filter import is_test_path
|
||||
|
||||
|
||||
def build_router_plan(route_result) -> dict[str, Any]:
|
||||
query_plan = route_result.query_plan
|
||||
retrieval_spec = route_result.retrieval_spec
|
||||
constraints = route_result.retrieval_constraints
|
||||
layers = [str(item.layer_id) for item in list(retrieval_spec.layer_queries or [])]
|
||||
path_scope = list(getattr(retrieval_spec.filters, "path_scope", []) or [])
|
||||
return {
|
||||
"intent": route_result.intent,
|
||||
"sub_intent": query_plan.sub_intent,
|
||||
"graph_id": route_result.graph_id,
|
||||
"retrieval_profile": route_result.retrieval_profile,
|
||||
"conversation_mode": route_result.conversation_mode,
|
||||
"layers": layers,
|
||||
"symbol_kind_hint": query_plan.symbol_kind_hint,
|
||||
"symbol_candidates": list(query_plan.symbol_candidates or []),
|
||||
"keyword_hints": list(query_plan.keyword_hints or []),
|
||||
"path_hints": list(query_plan.path_hints or []),
|
||||
"path_scope": path_scope,
|
||||
"doc_scope_hints": list(query_plan.doc_scope_hints or []),
|
||||
"retrieval_constraints": {
|
||||
"include_globs": list(constraints.include_globs or []),
|
||||
"exclude_globs": list(constraints.exclude_globs or []),
|
||||
"prefer_globs": list(constraints.prefer_globs or []),
|
||||
"test_file_globs": list(constraints.test_file_globs or []),
|
||||
"test_symbol_patterns": list(constraints.test_symbol_patterns or []),
|
||||
"max_candidates": int(constraints.max_candidates),
|
||||
"fuzzy_symbol_search": {
|
||||
"enabled": bool(constraints.fuzzy_symbol_search.enabled),
|
||||
"max_distance": int(constraints.fuzzy_symbol_search.max_distance),
|
||||
"top_k": int(constraints.fuzzy_symbol_search.top_k),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def init_diagnostics(route_result, *, router_ms: int, include_retrieval: bool) -> dict[str, Any]:
|
||||
diagnostics = {
|
||||
"router_plan": build_router_plan(route_result),
|
||||
"execution": {
|
||||
"executed_layers": [],
|
||||
"retrieval_mode_by_layer": {},
|
||||
"top_k_by_layer": {},
|
||||
"filters_by_layer": {},
|
||||
"repo_scope": {"repo_id": None, "workspace_id": None},
|
||||
},
|
||||
"retrieval": None,
|
||||
"constraint_violations": [],
|
||||
"timings_ms": {
|
||||
"router": int(router_ms),
|
||||
"symbol_resolution": 0,
|
||||
"retrieval_total": 0,
|
||||
"retrieval_by_layer": {},
|
||||
"merge_rank": 0,
|
||||
"prompt_build": 0,
|
||||
"llm_call": 0,
|
||||
},
|
||||
"prompt": None,
|
||||
}
|
||||
if include_retrieval:
|
||||
diagnostics["retrieval"] = {
|
||||
"requests": [],
|
||||
"applied": [],
|
||||
"fallback": {"used": False, "reason": None},
|
||||
}
|
||||
return diagnostics
|
||||
|
||||
|
||||
def clone_diagnostics(diagnostics: dict[str, Any]) -> dict[str, Any]:
|
||||
return deepcopy(diagnostics)
|
||||
|
||||
|
||||
def apply_retrieval_report(diagnostics: dict[str, Any], report: dict[str, Any] | None) -> None:
|
||||
if not report:
|
||||
return
|
||||
execution = diagnostics.get("execution", {})
|
||||
execution["executed_layers"] = list(report.get("executed_layers", []) or [])
|
||||
execution["retrieval_mode_by_layer"] = dict(report.get("retrieval_mode_by_layer", {}) or {})
|
||||
execution["top_k_by_layer"] = dict(report.get("top_k_by_layer", {}) or {})
|
||||
execution["filters_by_layer"] = dict(report.get("filters_by_layer", {}) or {})
|
||||
retrieval = diagnostics.get("retrieval")
|
||||
if isinstance(retrieval, dict):
|
||||
retrieval["requests"] = list(report.get("requests", []) or [])
|
||||
retrieval["applied"] = list(report.get("applied", []) or [])
|
||||
retrieval["fallback"] = dict(report.get("fallback", {"used": False, "reason": None}) or {})
|
||||
timings = diagnostics.get("timings_ms", {})
|
||||
timings["retrieval_by_layer"] = dict(report.get("retrieval_by_layer_ms", {}) or {})
|
||||
|
||||
|
||||
def assign_repo_scope(
|
||||
diagnostics: dict[str, Any],
|
||||
rag_rows: list[dict[str, Any]],
|
||||
*,
|
||||
expected_repo_id: str | None = None,
|
||||
expected_workspace_id: str | None = None,
|
||||
) -> None:
|
||||
repo_ids = sorted({str(_row_repo_id(row)).strip() for row in rag_rows if str(_row_repo_id(row)).strip()})
|
||||
workspace_ids = sorted({str(_row_workspace_id(row)).strip() for row in rag_rows if str(_row_workspace_id(row)).strip()})
|
||||
repo_id = expected_repo_id or (repo_ids[0] if len(repo_ids) == 1 else None)
|
||||
workspace_id = expected_workspace_id or (workspace_ids[0] if len(workspace_ids) == 1 else None)
|
||||
diagnostics["execution"]["repo_scope"] = {
|
||||
"repo_id": repo_id,
|
||||
"workspace_id": workspace_id,
|
||||
"returned_repo_ids": repo_ids,
|
||||
"returned_workspace_ids": workspace_ids,
|
||||
}
|
||||
|
||||
|
||||
def validate_constraints(
|
||||
route_result,
|
||||
rag_rows: list[dict[str, Any]],
|
||||
symbol_resolution: dict[str, Any],
|
||||
*,
|
||||
expected_repo_id: str | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
violations: list[dict[str, Any]] = []
|
||||
query_plan = route_result.query_plan
|
||||
retrieval_spec = route_result.retrieval_spec
|
||||
path_scope = list(getattr(retrieval_spec.filters, "path_scope", []) or [])
|
||||
row_paths = {str(row.get("path") or "") for row in rag_rows}
|
||||
if query_plan.sub_intent == "OPEN_FILE" and path_scope:
|
||||
if not any(path in row_paths for path in path_scope):
|
||||
violations.append(
|
||||
_violation(
|
||||
"PATH_SCOPE_NOT_SATISFIED",
|
||||
"error",
|
||||
{"path_scope": path_scope, "returned_paths": sorted(row_paths)},
|
||||
"Use exact path_scope retrieval and disable broad fallback for OPEN_FILE.",
|
||||
)
|
||||
)
|
||||
requested_layers = [str(item.layer_id) for item in list(retrieval_spec.layer_queries or [])]
|
||||
row_layers = {str(row.get("layer") or "") for row in rag_rows}
|
||||
for layer in requested_layers:
|
||||
if layer not in row_layers:
|
||||
violations.append(
|
||||
_violation(
|
||||
"LAYER_MISSING",
|
||||
"warn",
|
||||
{"layer": layer, "requested_layers": requested_layers},
|
||||
"Increase top_k for this layer or relax constraints for retrieval.",
|
||||
)
|
||||
)
|
||||
exclude_globs = list(route_result.retrieval_constraints.exclude_globs or [])
|
||||
if _tests_excluded(exclude_globs):
|
||||
test_paths = sorted({path for path in row_paths if is_test_path(path)})
|
||||
if test_paths:
|
||||
violations.append(
|
||||
_violation(
|
||||
"TESTS_EXCLUSION_VIOLATED",
|
||||
"warn",
|
||||
{"exclude_globs": exclude_globs, "test_paths": test_paths},
|
||||
"Tighten test exclude globs and apply them at pre-rank filtering stage.",
|
||||
)
|
||||
)
|
||||
if expected_repo_id:
|
||||
mismatched = sorted(
|
||||
{
|
||||
repo_id
|
||||
for repo_id in {str(_row_repo_id(row)).strip() for row in rag_rows if str(_row_repo_id(row)).strip()}
|
||||
if repo_id != expected_repo_id
|
||||
}
|
||||
)
|
||||
if mismatched:
|
||||
violations.append(
|
||||
_violation(
|
||||
"REPO_SCOPE_MISMATCH",
|
||||
"error",
|
||||
{"expected_repo_id": expected_repo_id, "returned_repo_ids": mismatched},
|
||||
"Use rag_session_id/repo scope that matches the expected repository.",
|
||||
)
|
||||
)
|
||||
symbol_candidates = list(query_plan.symbol_candidates or [])
|
||||
status = str(symbol_resolution.get("status") or "not_requested")
|
||||
if query_plan.sub_intent == "EXPLAIN" and symbol_candidates and status != "resolved":
|
||||
violations.append(
|
||||
_violation(
|
||||
"SYMBOL_RESOLUTION_FAILED",
|
||||
"warn",
|
||||
{"status": status, "symbol_candidates": symbol_candidates},
|
||||
"Increase symbol layer budget or improve fuzzy symbol resolution settings.",
|
||||
)
|
||||
)
|
||||
return violations
|
||||
|
||||
|
||||
def build_prompt_diagnostics(
|
||||
*,
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
rag_rows: list[dict[str, Any]],
|
||||
prompt_template_id: str | None = None,
|
||||
system_prompt_version: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
evidence_chars = sum(len(str(row.get("content") or "")) for row in rag_rows)
|
||||
tokens_in_estimate = max(1, int(math.ceil((len(system_prompt) + len(user_prompt)) / 4)))
|
||||
per_layer: dict[str, dict[str, Any]] = {}
|
||||
for row in rag_rows:
|
||||
layer = str(row.get("layer") or "")
|
||||
bucket = per_layer.setdefault(layer, {"layer": layer, "count": 0, "paths": set()})
|
||||
bucket["count"] += 1
|
||||
bucket["paths"].add(str(row.get("path") or ""))
|
||||
evidence_summary = [
|
||||
{"layer": item["layer"], "count": item["count"], "unique_paths": len(item["paths"])}
|
||||
for item in per_layer.values()
|
||||
]
|
||||
return {
|
||||
"prompt_stats": {
|
||||
"tokens_in_estimate": tokens_in_estimate,
|
||||
"evidence_rows": len(rag_rows),
|
||||
"evidence_chars": evidence_chars,
|
||||
},
|
||||
"evidence_summary": evidence_summary,
|
||||
"prompt_template_id": prompt_template_id,
|
||||
"system_prompt_version": system_prompt_version,
|
||||
}
|
||||
|
||||
|
||||
def _tests_excluded(exclude_globs: list[str]) -> bool:
|
||||
tokens = ("tests", "test", "__tests__", "mocks", "fixtures", "stubs", "conftest")
|
||||
return any(any(token in str(item).lower() for token in tokens) for item in exclude_globs)
|
||||
|
||||
|
||||
def _violation(violation_type: str, severity: str, details: dict[str, Any], suggested_fix: str) -> dict[str, Any]:
|
||||
return {
|
||||
"type": violation_type,
|
||||
"severity": severity,
|
||||
"details": details,
|
||||
"suggested_fix": suggested_fix,
|
||||
}
|
||||
|
||||
|
||||
def _row_repo_id(row: dict[str, Any]) -> str | None:
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
return metadata.get("repo_id")
|
||||
|
||||
|
||||
def _row_workspace_id(row: dict[str, Any]) -> str | None:
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
return metadata.get("workspace_id")
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from tests.pipeline_setup.env_loader import load_pipeline_setup_env
|
||||
|
||||
|
||||
class PipelineEnvLoader:
|
||||
def __init__(self, test_root: Path) -> None:
|
||||
self._test_root = test_root
|
||||
|
||||
def load(self) -> list[Path]:
|
||||
loaded: list[Path] = []
|
||||
test_env_path = self._test_root / ".env.test"
|
||||
if test_env_path.is_file():
|
||||
self._apply_file(test_env_path)
|
||||
loaded.append(test_env_path)
|
||||
loaded.extend(load_pipeline_setup_env(start_dir=self._test_root))
|
||||
return loaded
|
||||
|
||||
def _apply_file(self, path: Path) -> None:
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, raw_value = line.split("=", 1)
|
||||
name = key.removeprefix("export ").strip()
|
||||
if not name:
|
||||
continue
|
||||
os.environ[name] = self._normalize_value(raw_value.strip())
|
||||
|
||||
def _normalize_value(self, value: str) -> str:
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
|
||||
return value[1:-1]
|
||||
return value
|
||||
@@ -0,0 +1,64 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from app.modules.shared.gigachat.client import GigaChatClient
|
||||
from app.modules.shared.gigachat.settings import GigaChatSettings
|
||||
from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.answer_quality import AnswerQualityController
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.prompt_payload_builder import PromptPayloadBuilder
|
||||
|
||||
|
||||
class GigaChatAnswerer:
|
||||
_SYSTEM_PROMPT = "Ты технический ассистент по коду и документации."
|
||||
_PROMPT_TEMPLATE_ID = "pipeline_intent_rag_v1"
|
||||
_SYSTEM_PROMPT_VERSION = "v1"
|
||||
_TASK_PROMPT = "Дай краткий, практичный ответ по контексту. Если данных мало, явно скажи об этом."
|
||||
|
||||
def __init__(self) -> None:
|
||||
settings = GigaChatSettings.from_env()
|
||||
if not settings.credentials:
|
||||
raise ValueError("GIGACHAT_TOKEN is required for full_chain mode")
|
||||
self._client = GigaChatClient(settings, GigaChatTokenProvider(settings))
|
||||
self._payload_builder = PromptPayloadBuilder()
|
||||
self._quality = AnswerQualityController()
|
||||
|
||||
def answer(
|
||||
self,
|
||||
query: str,
|
||||
rag_rows: list[dict],
|
||||
*,
|
||||
prompt_template: dict | None = None,
|
||||
sub_intent: str | None = None,
|
||||
) -> str:
|
||||
prompt_payload = self.build_prompt_payload(
|
||||
query,
|
||||
rag_rows,
|
||||
prompt_template=prompt_template,
|
||||
sub_intent=sub_intent,
|
||||
)
|
||||
return self.answer_from_payload(prompt_payload)
|
||||
|
||||
def build_prompt_payload(
|
||||
self,
|
||||
query: str,
|
||||
rag_rows: list[dict],
|
||||
*,
|
||||
prompt_template: dict | None = None,
|
||||
sub_intent: str | None = None,
|
||||
) -> dict:
|
||||
return self._payload_builder.build(
|
||||
query,
|
||||
rag_rows,
|
||||
prompt_template=prompt_template,
|
||||
sub_intent=sub_intent,
|
||||
default_system_prompt=self._SYSTEM_PROMPT,
|
||||
default_task_prompt=self._TASK_PROMPT,
|
||||
default_template_id=self._PROMPT_TEMPLATE_ID,
|
||||
system_prompt_version=self._SYSTEM_PROMPT_VERSION,
|
||||
)
|
||||
|
||||
def answer_from_payload(self, payload: dict) -> str:
|
||||
answer = self._client.complete(
|
||||
system_prompt=str(payload.get("system_prompt") or self._SYSTEM_PROMPT),
|
||||
user_prompt=str(payload.get("user_prompt") or ""),
|
||||
)
|
||||
return self._quality.refine(answer, sub_intent=str(payload.get("sub_intent") or "").strip() or None)
|
||||
@@ -0,0 +1,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
class LlmPromptCatalogLoader:
|
||||
def load(self, path: Path) -> dict:
|
||||
payload = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError(f"Invalid prompt catalog: expected mapping, got {type(payload).__name__}")
|
||||
default = self._normalize_item("default", payload.get("default"))
|
||||
intents_raw = payload.get("intents", {})
|
||||
if not isinstance(intents_raw, dict):
|
||||
raise ValueError("Invalid prompt catalog: `intents` must be a mapping")
|
||||
intents: dict[str, dict] = {}
|
||||
for intent, item in intents_raw.items():
|
||||
intent_key = str(intent).strip()
|
||||
if not intent_key:
|
||||
continue
|
||||
intents[intent_key] = self._normalize_intent(intent_key, item)
|
||||
return {"default": default, "intents": intents}
|
||||
|
||||
def select_for_intent(self, catalog: dict, intent: str) -> dict:
|
||||
return self.select(catalog, intent=intent)
|
||||
|
||||
def select(self, catalog: dict, *, intent: str, sub_intent: str | None = None) -> dict:
|
||||
intents = dict(catalog.get("intents") or {})
|
||||
intent_entry = dict(intents.get(intent) or {})
|
||||
sub_intents = dict(intent_entry.get("sub_intents") or {})
|
||||
if sub_intent and sub_intent in sub_intents:
|
||||
return dict(sub_intents[sub_intent])
|
||||
if intent_entry:
|
||||
prompt = dict(intent_entry.get("prompt") or {})
|
||||
if prompt:
|
||||
return prompt
|
||||
return dict(catalog.get("default") or {})
|
||||
|
||||
def _normalize_intent(self, key: str, raw: object) -> dict:
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"Invalid prompt catalog item `{key}`: expected object")
|
||||
prompt = self._normalize_item(key, raw)
|
||||
sub_intents_raw = raw.get("sub_intents", {})
|
||||
if sub_intents_raw is None:
|
||||
sub_intents_raw = {}
|
||||
if not isinstance(sub_intents_raw, dict):
|
||||
raise ValueError(f"Invalid prompt catalog item `{key}`: `sub_intents` must be a mapping")
|
||||
sub_intents: dict[str, dict] = {}
|
||||
for sub_intent, item in sub_intents_raw.items():
|
||||
sub_key = str(sub_intent).strip()
|
||||
if not sub_key:
|
||||
continue
|
||||
sub_intents[sub_key] = self._normalize_item(f"{key}.{sub_key}", item)
|
||||
return {"prompt": prompt, "sub_intents": sub_intents}
|
||||
|
||||
def _normalize_item(self, key: str, raw: object) -> dict:
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"Invalid prompt catalog item `{key}`: expected object")
|
||||
system_prompt = str(raw.get("system_prompt", "")).strip()
|
||||
task_prompt = str(raw.get("task_prompt", "")).strip()
|
||||
template_id = str(raw.get("template_id", "")).strip() or f"intent_{key.lower()}_v1"
|
||||
if not system_prompt:
|
||||
raise ValueError(f"Invalid prompt catalog item `{key}`: `system_prompt` is required")
|
||||
if not task_prompt:
|
||||
raise ValueError(f"Invalid prompt catalog item `{key}`: `task_prompt` is required")
|
||||
return {
|
||||
"template_id": template_id,
|
||||
"system_prompt": system_prompt,
|
||||
"task_prompt": task_prompt,
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.result_views import PipelineDiagnosticsBuilder, PipelineSummaryBuilder
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PhraseCase:
|
||||
case_id: str
|
||||
text: str
|
||||
rag_session_id: str | None = None
|
||||
tags: list[str] = field(default_factory=list)
|
||||
expected_intent: str | None = None
|
||||
expect_non_empty_rag: bool = True
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PipelineResult:
|
||||
case: PhraseCase
|
||||
mode: str
|
||||
run_started_at: datetime
|
||||
rag_session_id: str | None
|
||||
intent: str
|
||||
graph_id: str
|
||||
conversation_mode: str
|
||||
query: str
|
||||
symbol_resolution: dict
|
||||
rag_rows: list[dict] = field(default_factory=list)
|
||||
llm_answer: str | None = None
|
||||
diagnostics: dict = field(default_factory=dict)
|
||||
steps: list[dict] = field(default_factory=list)
|
||||
|
||||
def to_record(self) -> dict:
|
||||
diagnostics = PipelineDiagnosticsBuilder().build(self)
|
||||
summary = PipelineSummaryBuilder().build(self, diagnostics)
|
||||
steps = self.steps or self._default_steps(diagnostics)
|
||||
return {
|
||||
"case_id": self.case.case_id,
|
||||
"text": self.case.text,
|
||||
"mode": self.mode,
|
||||
"run_started_at": self.run_started_at.isoformat(timespec="seconds"),
|
||||
"rag_session_id": self.rag_session_id,
|
||||
"expected_intent": self.case.expected_intent,
|
||||
"actual_intent": self.intent,
|
||||
"graph_id": self.graph_id,
|
||||
"conversation_mode": self.conversation_mode,
|
||||
"query": self.query,
|
||||
"symbol_resolution": self.symbol_resolution,
|
||||
"rag_count": len(self.rag_rows),
|
||||
"rag_rows": self.rag_rows,
|
||||
"llm_answer": self.llm_answer,
|
||||
"summary": summary,
|
||||
"diagnostics": diagnostics,
|
||||
"run_info": {
|
||||
"case_id": self.case.case_id,
|
||||
"mode": self.mode,
|
||||
"run_started_at": self.run_started_at.isoformat(timespec="seconds"),
|
||||
"rag_session_id": self.rag_session_id,
|
||||
"expected_intent": self.case.expected_intent,
|
||||
"actual_intent": self.intent,
|
||||
"graph_id": self.graph_id,
|
||||
"conversation_mode": self.conversation_mode,
|
||||
},
|
||||
"input_request": {
|
||||
"text": self.case.text,
|
||||
"normalized_query": self.query,
|
||||
},
|
||||
"steps": steps,
|
||||
}
|
||||
|
||||
def _default_steps(self, diagnostics: dict) -> list[dict]:
|
||||
timings = dict(diagnostics.get("timings_ms") or {})
|
||||
retrieval_diag = diagnostics.get("retrieval")
|
||||
router_step = {
|
||||
"step": "intent_router",
|
||||
"input": {"query": self.case.text},
|
||||
"output": {
|
||||
"intent": self.intent,
|
||||
"graph_id": self.graph_id,
|
||||
"conversation_mode": self.conversation_mode,
|
||||
"query": self.query,
|
||||
},
|
||||
"diagnostics": {
|
||||
"router_plan": diagnostics.get("router_plan"),
|
||||
"timings_ms": {"router": timings.get("router", 0)},
|
||||
},
|
||||
}
|
||||
if self.mode == "router_only":
|
||||
return [router_step]
|
||||
retrieval_step = {
|
||||
"step": "retrieval",
|
||||
"input": {"query": self.query},
|
||||
"output": {
|
||||
"symbol_resolution": self.symbol_resolution,
|
||||
"rag_count": len(self.rag_rows),
|
||||
"rag_rows": self.rag_rows,
|
||||
},
|
||||
"diagnostics": {
|
||||
"execution": diagnostics.get("execution"),
|
||||
"retrieval": retrieval_diag,
|
||||
"constraint_violations": diagnostics.get("constraint_violations"),
|
||||
"timings_ms": {
|
||||
"symbol_resolution": timings.get("symbol_resolution", 0),
|
||||
"retrieval_total": timings.get("retrieval_total", 0),
|
||||
"retrieval_by_layer": timings.get("retrieval_by_layer", {}),
|
||||
"merge_rank": timings.get("merge_rank", 0),
|
||||
},
|
||||
},
|
||||
}
|
||||
if self.mode == "router_rag":
|
||||
return [router_step, retrieval_step]
|
||||
llm_step = {
|
||||
"step": "llm_answer",
|
||||
"input": {"query": self.query, "rag_count": len(self.rag_rows)},
|
||||
"output": {"llm_answer": self.llm_answer},
|
||||
"diagnostics": {
|
||||
"prompt": diagnostics.get("prompt"),
|
||||
"timings_ms": {
|
||||
"prompt_build": timings.get("prompt_build", 0),
|
||||
"llm_call": timings.get("llm_call", 0),
|
||||
},
|
||||
},
|
||||
}
|
||||
return [router_step, retrieval_step, llm_step]
|
||||
@@ -0,0 +1,45 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase
|
||||
|
||||
|
||||
class PhraseCatalogLoader:
|
||||
def load(self, path: Path) -> list[PhraseCase]:
|
||||
payload = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError(f"Invalid phrases file: expected mapping, got {type(payload).__name__}")
|
||||
items = payload.get("phrases", [])
|
||||
if not isinstance(items, list):
|
||||
raise ValueError("Invalid phrases file: `phrases` must be a list")
|
||||
cases = [self._to_case(raw) for raw in items]
|
||||
if not cases:
|
||||
raise ValueError("No phrase cases found in phrases file")
|
||||
return cases
|
||||
|
||||
def filter_by_tag(self, cases: list[PhraseCase], tag: str) -> list[PhraseCase]:
|
||||
return [case for case in cases if tag in case.tags]
|
||||
|
||||
def _to_case(self, raw: object) -> PhraseCase:
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError("Invalid phrase item: expected object")
|
||||
case_id = str(raw.get("id", "")).strip()
|
||||
text = str(raw.get("text", "")).strip()
|
||||
if not case_id:
|
||||
raise ValueError("Invalid phrase item: `id` is required")
|
||||
if not text:
|
||||
raise ValueError(f"Invalid phrase item `{case_id}`: `text` is required")
|
||||
tags = [str(item).strip() for item in raw.get("tags", []) if str(item).strip()]
|
||||
expected_intent = raw.get("expected_intent")
|
||||
rag_session_id = str(raw.get("rag_session_id", "")).strip() or None
|
||||
return PhraseCase(
|
||||
case_id=case_id,
|
||||
text=text,
|
||||
rag_session_id=rag_session_id,
|
||||
tags=tags,
|
||||
expected_intent=str(expected_intent).strip() if expected_intent else None,
|
||||
expect_non_empty_rag=bool(raw.get("expect_non_empty_rag", True)),
|
||||
)
|
||||
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.env_bootstrap import PipelineEnvLoader
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PipelineRunConfig:
|
||||
mode: str
|
||||
test_name: str
|
||||
tag: str
|
||||
phrases_path: Path
|
||||
test_results_dir: Path
|
||||
forced_rag_session_id: str | None
|
||||
default_rag_session_id: str | None
|
||||
reindex_repo_path: Path | None
|
||||
reindex_project_id: str | None
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, mode: str, test_name: str, test_root: Path) -> "PipelineRunConfig":
|
||||
PipelineEnvLoader(test_root).load()
|
||||
mode_tag = {
|
||||
"router_only": "router_only",
|
||||
"router_rag": "router_rag",
|
||||
"full_chain": "full_chain",
|
||||
}[mode]
|
||||
forced_sid = os.getenv("INTENT_PIPELINE_FORCE_RAG_SESSION_ID", "").strip() or None
|
||||
default_sid = os.getenv("INTENT_PIPELINE_RAG_SESSION_ID", "").strip() or None
|
||||
raw_reindex_path = os.getenv("INTENT_PIPELINE_REINDEX_REPO_PATH", "").strip()
|
||||
reindex_repo_path = Path(raw_reindex_path).expanduser().resolve() if raw_reindex_path else None
|
||||
reindex_project_id = os.getenv("INTENT_PIPELINE_REINDEX_PROJECT_ID", "").strip() or None
|
||||
return cls(
|
||||
mode=mode,
|
||||
test_name=test_name,
|
||||
tag=mode_tag,
|
||||
phrases_path=test_root / "fixtures" / "phrases.yaml",
|
||||
test_results_dir=test_root.parent / "test_results",
|
||||
forced_rag_session_id=forced_sid,
|
||||
default_rag_session_id=default_sid,
|
||||
reindex_repo_path=reindex_repo_path,
|
||||
reindex_project_id=reindex_project_id,
|
||||
)
|
||||
|
||||
|
||||
def mode_enabled(mode: str, test_root: Path) -> bool:
|
||||
PipelineEnvLoader(test_root).load()
|
||||
if mode == "router_only":
|
||||
return True
|
||||
if mode == "router_rag":
|
||||
return _truthy(os.getenv("RUN_INTENT_PIPELINE_ROUTER_RAG") or os.getenv("RUN_INTENT_ROUTER_RAG_PIPELINE"))
|
||||
if mode == "full_chain":
|
||||
return _truthy(os.getenv("RUN_INTENT_PIPELINE_FULL_CHAIN"))
|
||||
return False
|
||||
|
||||
|
||||
def _truthy(value: str | None) -> bool:
|
||||
return str(value or "").strip().lower() in {"1", "true", "yes", "on"}
|
||||
@@ -0,0 +1,171 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from difflib import get_close_matches
|
||||
from time import perf_counter
|
||||
|
||||
from app.modules.rag.intent_router_v2 import ConversationState, IntentRouterV2
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.diagnostics import (
|
||||
apply_retrieval_report,
|
||||
assign_repo_scope,
|
||||
init_diagnostics,
|
||||
validate_constraints,
|
||||
)
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase, PipelineResult
|
||||
from tests.unit_tests.rag.intent_router_testkit import repo_context
|
||||
|
||||
|
||||
class RouterOnlyRunner:
|
||||
def __init__(self, started_at: datetime, router: IntentRouterV2 | None = None) -> None:
|
||||
self._started_at = started_at
|
||||
self._router = router or IntentRouterV2()
|
||||
|
||||
def run_case(self, case: PhraseCase) -> PipelineResult:
|
||||
router_started = perf_counter()
|
||||
route_result = self._router.route(case.text, ConversationState(), repo_context())
|
||||
router_ms = _ms(router_started)
|
||||
query = route_result.query_plan.normalized or route_result.query_plan.raw or case.text
|
||||
diagnostics = init_diagnostics(route_result, router_ms=router_ms, include_retrieval=False)
|
||||
return PipelineResult(
|
||||
case=case,
|
||||
mode="router_only",
|
||||
run_started_at=self._started_at,
|
||||
rag_session_id=case.rag_session_id,
|
||||
intent=route_result.intent,
|
||||
graph_id=route_result.graph_id,
|
||||
conversation_mode=route_result.conversation_mode,
|
||||
query=query,
|
||||
symbol_resolution=route_result.symbol_resolution.model_dump(),
|
||||
diagnostics=diagnostics,
|
||||
)
|
||||
|
||||
|
||||
class IntentRouterRagPipelineRunner:
|
||||
def __init__(
|
||||
self,
|
||||
started_at: datetime,
|
||||
rag_adapter,
|
||||
session_resolver,
|
||||
router: IntentRouterV2 | None = None,
|
||||
) -> None:
|
||||
self._started_at = started_at
|
||||
self._rag_adapter = rag_adapter
|
||||
self._session_resolver = session_resolver
|
||||
self._router = router or IntentRouterV2()
|
||||
|
||||
def run_case(self, case: PhraseCase) -> PipelineResult:
|
||||
router_started = perf_counter()
|
||||
route_result = self._router.route(case.text, ConversationState(), repo_context())
|
||||
router_ms = _ms(router_started)
|
||||
diagnostics = init_diagnostics(route_result, router_ms=router_ms, include_retrieval=True)
|
||||
rag_query = route_result.query_plan.normalized or route_result.query_plan.raw or case.text
|
||||
rag_session_id = self._session_resolver.resolve(case)
|
||||
retrieval_started = perf_counter()
|
||||
rag_rows = self._retrieve_rows(
|
||||
rag_session_id=rag_session_id,
|
||||
rag_query=rag_query,
|
||||
route_result=route_result,
|
||||
)
|
||||
diagnostics["timings_ms"]["retrieval_total"] = _ms(retrieval_started)
|
||||
consume_report = getattr(self._rag_adapter, "consume_retrieval_report", None)
|
||||
retrieval_report = consume_report() if callable(consume_report) else None
|
||||
apply_retrieval_report(diagnostics, retrieval_report)
|
||||
symbol_started = perf_counter()
|
||||
symbol_resolution = self._resolve_symbol(route_result.symbol_resolution.model_dump(), rag_rows)
|
||||
diagnostics["timings_ms"]["symbol_resolution"] = _ms(symbol_started)
|
||||
if (
|
||||
route_result.query_plan.sub_intent == "EXPLAIN"
|
||||
and list(route_result.query_plan.symbol_candidates or [])
|
||||
and str(symbol_resolution.get("status") or "") != "resolved"
|
||||
):
|
||||
rag_rows = []
|
||||
if isinstance(diagnostics.get("retrieval"), dict):
|
||||
diagnostics["retrieval"]["fallback"] = {"used": False, "reason": None}
|
||||
merge_started = perf_counter()
|
||||
rag_rows = self._rag_adapter.hydrate_resolved_symbol_sources(
|
||||
rag_session_id=rag_session_id,
|
||||
base_query=rag_query,
|
||||
rag_rows=rag_rows,
|
||||
symbol_resolution=symbol_resolution,
|
||||
retrieval_spec=route_result.retrieval_spec,
|
||||
retrieval_constraints=route_result.retrieval_constraints,
|
||||
)
|
||||
rag_rows = self._rag_adapter.force_symbol_context_c0(
|
||||
rag_session_id=rag_session_id,
|
||||
rag_rows=rag_rows,
|
||||
symbol_resolution=symbol_resolution,
|
||||
limit=20,
|
||||
)
|
||||
diagnostics["timings_ms"]["merge_rank"] = _ms(merge_started)
|
||||
expected_repo_id = str(getattr(route_result.retrieval_spec.filters, "repo_id", "") or "").strip() or None
|
||||
assign_repo_scope(diagnostics, rag_rows, expected_repo_id=expected_repo_id)
|
||||
diagnostics["constraint_violations"] = validate_constraints(
|
||||
route_result,
|
||||
rag_rows,
|
||||
symbol_resolution,
|
||||
expected_repo_id=expected_repo_id,
|
||||
)
|
||||
return PipelineResult(
|
||||
case=case,
|
||||
mode="router_rag",
|
||||
run_started_at=self._started_at,
|
||||
rag_session_id=rag_session_id,
|
||||
intent=route_result.intent,
|
||||
graph_id=route_result.graph_id,
|
||||
conversation_mode=route_result.conversation_mode,
|
||||
query=rag_query,
|
||||
rag_rows=rag_rows,
|
||||
symbol_resolution=symbol_resolution,
|
||||
diagnostics=diagnostics,
|
||||
)
|
||||
|
||||
def _resolve_symbol(self, initial: dict, rag_rows: list[dict]) -> dict:
|
||||
status = str(initial.get("status") or "not_requested")
|
||||
if status != "pending":
|
||||
return initial
|
||||
candidates = [str(item) for item in initial.get("alternatives", []) if str(item).strip()]
|
||||
c1_titles = list(dict.fromkeys(
|
||||
str(row.get("title") or "").strip()
|
||||
for row in rag_rows
|
||||
if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG" and str(row.get("title") or "").strip()
|
||||
))
|
||||
if not c1_titles:
|
||||
initial["status"] = "not_found"
|
||||
return initial
|
||||
exact = next((title for title in c1_titles if title in candidates), None)
|
||||
if exact:
|
||||
return {"status": "resolved", "resolved_symbol": exact, "alternatives": c1_titles[:5], "confidence": 0.99}
|
||||
fuzzy_hits: list[str] = []
|
||||
for candidate in candidates:
|
||||
fuzzy_hits.extend(get_close_matches(candidate, c1_titles, n=3, cutoff=0.7))
|
||||
fuzzy_hits = list(dict.fromkeys(fuzzy_hits))
|
||||
if len(fuzzy_hits) == 1:
|
||||
return {"status": "resolved", "resolved_symbol": fuzzy_hits[0], "alternatives": c1_titles[:5], "confidence": 0.82}
|
||||
if len(fuzzy_hits) > 1:
|
||||
return {"status": "ambiguous", "resolved_symbol": None, "alternatives": fuzzy_hits[:5], "confidence": 0.55}
|
||||
return {"status": "not_found", "resolved_symbol": None, "alternatives": c1_titles[:5], "confidence": 0.0}
|
||||
|
||||
def _retrieve_rows(self, *, rag_session_id: str, rag_query: str, route_result) -> list[dict]:
|
||||
path_scope = list(getattr(route_result.retrieval_spec.filters, "path_scope", []) or [])
|
||||
expected_repo_id = str(getattr(route_result.retrieval_spec.filters, "repo_id", "") or "").strip() or None
|
||||
if route_result.query_plan.sub_intent == "OPEN_FILE" and path_scope:
|
||||
return self._rag_adapter.retrieve_exact_files(
|
||||
rag_session_id=rag_session_id,
|
||||
repo_id=expected_repo_id,
|
||||
paths=path_scope,
|
||||
layers=["C0_SOURCE_CHUNKS"],
|
||||
limit=200,
|
||||
query=rag_query,
|
||||
ranking_profile=str(getattr(route_result.retrieval_spec, "rerank_profile", "") or ""),
|
||||
)
|
||||
return self._rag_adapter.retrieve_with_plan(
|
||||
rag_session_id,
|
||||
rag_query,
|
||||
route_result.retrieval_spec,
|
||||
route_result.retrieval_constraints,
|
||||
query_plan=route_result.query_plan,
|
||||
)
|
||||
|
||||
|
||||
def _ms(started: float) -> int:
|
||||
return int((perf_counter() - started) * 1000)
|
||||
@@ -0,0 +1,148 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.diagnostics import build_prompt_diagnostics
|
||||
|
||||
|
||||
class PromptPayloadBuilder:
|
||||
def build(
|
||||
self,
|
||||
query: str,
|
||||
rag_rows: list[dict],
|
||||
*,
|
||||
prompt_template: dict | None = None,
|
||||
sub_intent: str | None = None,
|
||||
default_system_prompt: str,
|
||||
default_task_prompt: str,
|
||||
default_template_id: str,
|
||||
system_prompt_version: str,
|
||||
) -> dict:
|
||||
template = dict(prompt_template or {})
|
||||
system_prompt = str(template.get("system_prompt") or default_system_prompt)
|
||||
task_prompt = str(template.get("task_prompt") or default_task_prompt)
|
||||
template_id = str(template.get("template_id") or default_template_id)
|
||||
context = RagContextFormatter().format(rag_rows, sub_intent=sub_intent)
|
||||
user_prompt = (
|
||||
f"Вопрос пользователя:\n{query}\n\n"
|
||||
f"Правила ответа:\n{task_prompt}\n\n"
|
||||
f"Как интерпретировать слои RAG:\n{_LAYER_GUIDE}\n\n"
|
||||
f"Контекст RAG:\n{context}\n\n"
|
||||
"Ответь пользователю естественным инженерным языком."
|
||||
)
|
||||
return {
|
||||
"system_prompt": system_prompt,
|
||||
"user_prompt": user_prompt,
|
||||
"sub_intent": str(sub_intent or "").strip() or None,
|
||||
"diagnostics": build_prompt_diagnostics(
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
rag_rows=rag_rows,
|
||||
prompt_template_id=template_id,
|
||||
system_prompt_version=system_prompt_version,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
class RagContextFormatter:
|
||||
_DEFAULT_LIMIT = 6
|
||||
_QUOTAS = {
|
||||
"OPEN_FILE": {"C0_SOURCE_CHUNKS": 4, "C1_SYMBOL_CATALOG": 1, "C4_SEMANTIC_ROLES": 1},
|
||||
"EXPLAIN": {
|
||||
"C1_SYMBOL_CATALOG": 1,
|
||||
"C0_SOURCE_CHUNKS": 2,
|
||||
"C2_DEPENDENCY_GRAPH": 1,
|
||||
"C4_SEMANTIC_ROLES": 1,
|
||||
"C3_ENTRYPOINTS": 1,
|
||||
},
|
||||
"EXPLAIN_LOCAL": {"C0_SOURCE_CHUNKS": 3, "C1_SYMBOL_CATALOG": 1, "C2_DEPENDENCY_GRAPH": 1, "C4_SEMANTIC_ROLES": 1},
|
||||
"FIND_TESTS": {"C1_SYMBOL_CATALOG": 1, "C2_DEPENDENCY_GRAPH": 2, "C0_SOURCE_CHUNKS": 2, "C4_SEMANTIC_ROLES": 1},
|
||||
"FIND_ENTRYPOINTS": {"C3_ENTRYPOINTS": 3, "C0_SOURCE_CHUNKS": 2, "C1_SYMBOL_CATALOG": 1},
|
||||
"TRACE_FLOW": {"C2_DEPENDENCY_GRAPH": 2, "C3_ENTRYPOINTS": 1, "C0_SOURCE_CHUNKS": 2, "C1_SYMBOL_CATALOG": 1},
|
||||
"ARCHITECTURE": {"C4_SEMANTIC_ROLES": 2, "C2_DEPENDENCY_GRAPH": 2, "C3_ENTRYPOINTS": 1, "C1_SYMBOL_CATALOG": 1},
|
||||
}
|
||||
|
||||
def format(self, rag_rows: list[dict], *, sub_intent: str | None = None) -> str:
|
||||
if not rag_rows:
|
||||
return "(пусто)"
|
||||
selected = self._select_rows(rag_rows, sub_intent=sub_intent)
|
||||
return "\n\n".join(_RagRowFormatter().format(row) for row in selected)
|
||||
|
||||
def _select_rows(self, rag_rows: list[dict], *, sub_intent: str | None) -> list[dict]:
|
||||
rows = self._prioritize_rows(rag_rows, sub_intent=sub_intent)
|
||||
quotas = dict(self._QUOTAS.get(str(sub_intent or "").strip().upper(), {}))
|
||||
if not quotas:
|
||||
return rows[: self._DEFAULT_LIMIT]
|
||||
selected: list[dict] = []
|
||||
counts = {layer: 0 for layer in quotas}
|
||||
for row in rows:
|
||||
layer = str(row.get("layer") or "")
|
||||
if layer not in quotas or counts[layer] >= quotas[layer]:
|
||||
continue
|
||||
selected.append(row)
|
||||
counts[layer] += 1
|
||||
if len(selected) >= self._DEFAULT_LIMIT:
|
||||
return selected[: self._DEFAULT_LIMIT]
|
||||
for row in rows:
|
||||
if row in selected:
|
||||
continue
|
||||
selected.append(row)
|
||||
if len(selected) >= self._DEFAULT_LIMIT:
|
||||
break
|
||||
return selected
|
||||
|
||||
def _prioritize_rows(self, rag_rows: list[dict], *, sub_intent: str | None) -> list[dict]:
|
||||
if str(sub_intent or "").strip().upper() != "FIND_ENTRYPOINTS":
|
||||
return list(rag_rows)
|
||||
ranked = sorted(
|
||||
enumerate(rag_rows),
|
||||
key=lambda item: (_entrypoint_priority(item[1]), item[0]),
|
||||
)
|
||||
return [row for _, row in ranked]
|
||||
|
||||
|
||||
class _RagRowFormatter:
|
||||
def format(self, row: dict) -> str:
|
||||
layer = str(row.get("layer") or "")
|
||||
path = str(row.get("path") or "")
|
||||
title = str(row.get("title") or "")
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
content = self._content(row, metadata)
|
||||
return f"- [{layer}] {path} | {title}\n{content}"
|
||||
|
||||
def _content(self, row: dict, metadata: dict) -> str:
|
||||
if str(row.get("layer") or "") != "C3_ENTRYPOINTS":
|
||||
return str(row.get("content") or "")[:1200]
|
||||
http_method = str(metadata.get("http_method") or "").strip()
|
||||
route_path = str(metadata.get("route_path") or "").strip()
|
||||
if not http_method or not route_path:
|
||||
return str(row.get("content") or "")[:1200]
|
||||
lines = [str(metadata.get("summary_text") or f"{http_method} {route_path}").strip()]
|
||||
declaring = str(metadata.get("declaring_symbol") or "").strip()
|
||||
handler = str(metadata.get("handler_symbol") or "").strip()
|
||||
decorator_text = str(metadata.get("decorator_text") or "").strip()
|
||||
if declaring:
|
||||
lines.append(f"Declared in: {declaring}")
|
||||
if handler:
|
||||
lines.append(f"Handler: {handler}")
|
||||
if decorator_text:
|
||||
lines.append(f"Decorator: {decorator_text}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _entrypoint_priority(row: dict) -> tuple[int, int]:
|
||||
if str(row.get("layer") or "") != "C3_ENTRYPOINTS":
|
||||
return (2, 0)
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
if str(metadata.get("http_method") or "").strip() and str(metadata.get("route_path") or "").strip():
|
||||
return (0, 0)
|
||||
if str(metadata.get("entry_type") or "").strip() == "http":
|
||||
return (1, 0)
|
||||
return (1, 1)
|
||||
|
||||
|
||||
_LAYER_GUIDE = (
|
||||
"- C0_SOURCE_CHUNKS: фактический код, это самый надёжный источник деталей реализации.\n"
|
||||
"- C1_SYMBOL_CATALOG: объявления символов, сигнатуры и базовая идентификация сущностей.\n"
|
||||
"- C2_DEPENDENCY_GRAPH: связи вызовов, зависимостей и потоков между сущностями.\n"
|
||||
"- C3_ENTRYPOINTS: подтверждённые точки входа. Если есть http_method и route_path, считай это сильным сигналом реального route.\n"
|
||||
"- C4_SEMANTIC_ROLES: эвристическое описание роли компонента; используй как вспомогательный сигнал и не ставь выше фактического кода."
|
||||
)
|
||||
@@ -0,0 +1,745 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from fnmatch import fnmatch
|
||||
import re
|
||||
from time import perf_counter
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.modules.rag.retrieval.test_filter import build_test_filters
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.modules.rag.persistence.repository import RagRepository
|
||||
|
||||
_KNOWN_FILE_EXTENSIONS = (
|
||||
".py",
|
||||
".md",
|
||||
".txt",
|
||||
".yaml",
|
||||
".yml",
|
||||
".json",
|
||||
".toml",
|
||||
".ini",
|
||||
)
|
||||
_SYMBOL_TOKEN_RE = re.compile(r"\b([A-Z][A-Za-z0-9_]{2,}|[a-z_][a-z0-9_]{2,})\b")
|
||||
_DEFAULT_SYMBOL_KINDS = {"class", "function", "method"}
|
||||
|
||||
|
||||
def is_file_path(path: str) -> bool:
|
||||
normalized = normalize_path(path)
|
||||
if not normalized:
|
||||
return False
|
||||
tail = normalized.rsplit("/", 1)[-1]
|
||||
if "." in tail and not tail.startswith(".") and not tail.endswith("."):
|
||||
return True
|
||||
return tail.lower().endswith(_KNOWN_FILE_EXTENSIONS)
|
||||
|
||||
|
||||
def normalize_path(path: str) -> str:
|
||||
normalized = str(path or "").strip().replace("\\", "/")
|
||||
while "//" in normalized:
|
||||
normalized = normalized.replace("//", "/")
|
||||
if normalized.startswith("./"):
|
||||
normalized = normalized[2:]
|
||||
return normalized
|
||||
|
||||
|
||||
def normalize_paths(paths: list[str]) -> list[str]:
|
||||
deduped: list[str] = []
|
||||
for value in paths:
|
||||
normalized = normalize_path(value)
|
||||
if normalized and normalized not in deduped:
|
||||
deduped.append(normalized)
|
||||
return deduped
|
||||
|
||||
|
||||
def extract_symbol_tokens(text: str) -> list[str]:
|
||||
tokens: list[str] = []
|
||||
for match in _SYMBOL_TOKEN_RE.finditer(str(text or "")):
|
||||
token = match.group(1).strip()
|
||||
if token and token not in tokens:
|
||||
tokens.append(token)
|
||||
return tokens[:8]
|
||||
|
||||
|
||||
class SessionEmbeddingDimensions:
|
||||
def __init__(self) -> None:
|
||||
self._cache: dict[str, int] = {}
|
||||
|
||||
def resolve(self, rag_session_id: str) -> int:
|
||||
if rag_session_id in self._cache:
|
||||
return self._cache[rag_session_id]
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.modules.shared.db import get_engine
|
||||
|
||||
with get_engine().connect() as conn:
|
||||
row = conn.execute(
|
||||
text(
|
||||
"""
|
||||
SELECT vector_dims(embedding) AS dim
|
||||
FROM rag_chunks
|
||||
WHERE rag_session_id = :sid
|
||||
LIMIT 1
|
||||
"""
|
||||
),
|
||||
{"sid": rag_session_id},
|
||||
).mappings().first()
|
||||
dim = int(row["dim"]) if row and row.get("dim") else 0
|
||||
self._cache[rag_session_id] = dim
|
||||
return dim
|
||||
|
||||
|
||||
class GlobPatternMapper:
|
||||
@staticmethod
|
||||
def to_prefixes(globs: list[str]) -> list[str]:
|
||||
result: list[str] = []
|
||||
for value in globs:
|
||||
item = str(value or "").strip()
|
||||
if not item:
|
||||
continue
|
||||
if item.endswith("/**"):
|
||||
prefix = item[:-3]
|
||||
elif "*" in item or "?" in item:
|
||||
continue
|
||||
else:
|
||||
prefix = item
|
||||
if prefix and prefix not in result:
|
||||
result.append(prefix)
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def to_like_patterns(globs: list[str]) -> list[str]:
|
||||
result: list[str] = []
|
||||
for value in globs:
|
||||
item = str(value or "").strip().lower()
|
||||
if not item or ("*" not in item and "?" not in item):
|
||||
continue
|
||||
sql_like = item.replace("**/", "%/").replace("**", "%").replace("*", "%").replace("?", "_")
|
||||
if sql_like not in result:
|
||||
result.append(sql_like)
|
||||
return result
|
||||
|
||||
|
||||
class RagRowDeduplicator:
|
||||
def dedupe(self, rows: list[dict]) -> list[dict]:
|
||||
result: list[dict] = []
|
||||
seen: set[tuple[Any, ...]] = set()
|
||||
for row in rows:
|
||||
key = self._key(row)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
def _key(self, row: dict) -> tuple[Any, ...]:
|
||||
layer = str(row.get("layer") or "")
|
||||
path = str(row.get("path") or "")
|
||||
span_start = row.get("span_start")
|
||||
span_end = row.get("span_end")
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
if layer == "C0_SOURCE_CHUNKS":
|
||||
return (layer, path, span_start, span_end)
|
||||
if layer == "C1_SYMBOL_CATALOG":
|
||||
return (layer, str(metadata.get("symbol_id") or ""), path, span_start, span_end)
|
||||
if layer == "C2_DEPENDENCY_GRAPH":
|
||||
return (layer, str(metadata.get("edge_id") or ""), path, span_start, span_end)
|
||||
return (layer, path, str(row.get("title") or ""), span_start, span_end)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class LayerRetrievalParams:
|
||||
rag_session_id: str
|
||||
query_text: str
|
||||
layer_id: str
|
||||
limit: int
|
||||
scope: list[str]
|
||||
exclude_prefixes: list[str] | None
|
||||
exclude_like: list[str] | None
|
||||
prefer_path_prefixes: list[str] | None
|
||||
prefer_like_patterns: list[str] | None
|
||||
dim: int
|
||||
include_tests: bool
|
||||
retrieval_mode_hint: str | None = None
|
||||
c1_kind_filter: set[str] | None = None
|
||||
allow_scope_relaxation: bool = True
|
||||
|
||||
|
||||
class LayerBudgetRetriever:
|
||||
def __init__(
|
||||
self,
|
||||
repository: RagRepository,
|
||||
dim_resolver: SessionEmbeddingDimensions,
|
||||
deduper: RagRowDeduplicator,
|
||||
) -> None:
|
||||
self._repository = repository
|
||||
self._dim_resolver = dim_resolver
|
||||
self._deduper = deduper
|
||||
self._explain_refiner = ExplainStrictRefiner()
|
||||
self._last_report: dict[str, Any] | None = None
|
||||
|
||||
def retrieve(
|
||||
self,
|
||||
query: str,
|
||||
retrieval_spec,
|
||||
retrieval_constraints,
|
||||
mapper: GlobPatternMapper,
|
||||
*,
|
||||
rag_session_id: str,
|
||||
query_plan=None,
|
||||
) -> list[dict]:
|
||||
filters = retrieval_spec.filters
|
||||
path_scope = list(getattr(filters, "path_scope", []) or [])
|
||||
test_policy = str(getattr(filters, "test_policy", "EXCLUDE") or "EXCLUDE")
|
||||
include_globs = list(getattr(retrieval_constraints, "include_globs", []) or [])
|
||||
exclude_globs = list(getattr(retrieval_constraints, "exclude_globs", []) or [])
|
||||
prefer_globs = list(getattr(retrieval_constraints, "prefer_globs", []) or [])
|
||||
include_prefixes = mapper.to_prefixes(include_globs)
|
||||
exclude_from_constraints = mapper.to_prefixes(exclude_globs)
|
||||
exclude_like_constraints = mapper.to_like_patterns(exclude_globs)
|
||||
prefer_path_prefixes = mapper.to_prefixes(prefer_globs)
|
||||
prefer_like_patterns = mapper.to_like_patterns(prefer_globs)
|
||||
test_file_globs = list(getattr(retrieval_constraints, "test_file_globs", []) or [])
|
||||
test_symbol_patterns = list(getattr(retrieval_constraints, "test_symbol_patterns", []) or [])
|
||||
exclude_prefixes, exclude_like = self._test_filters(test_policy)
|
||||
if exclude_from_constraints:
|
||||
exclude_prefixes = list(dict.fromkeys([*(exclude_prefixes or []), *exclude_from_constraints]))
|
||||
if exclude_like_constraints:
|
||||
exclude_like = list(dict.fromkeys([*(exclude_like or []), *exclude_like_constraints]))
|
||||
effective_scope = path_scope or include_prefixes
|
||||
effective_query = f"{query} {' '.join(test_symbol_patterns)}".strip() if test_symbol_patterns else query
|
||||
symbol_candidates = list(getattr(query_plan, "symbol_candidates", []) or [])
|
||||
symbol_kind_hint = str(getattr(query_plan, "symbol_kind_hint", "") or "").strip().lower()
|
||||
sub_intent = str(getattr(query_plan, "sub_intent", "") or "").strip().upper()
|
||||
c1_query = self._c1_query(symbol_candidates, query)
|
||||
c1_kind_filter = self._c1_kind_filter(symbol_candidates, symbol_kind_hint)
|
||||
strict_symbol_mode = bool(symbol_candidates) and sub_intent == "EXPLAIN"
|
||||
if strict_symbol_mode and c1_query:
|
||||
effective_query = c1_query
|
||||
dim = self._dim_resolver.resolve(rag_session_id)
|
||||
merged: list[dict] = []
|
||||
report = self._empty_report()
|
||||
report["fallback"]["reason"] = None
|
||||
for layer_query in list(getattr(retrieval_spec, "layer_queries", []) or []):
|
||||
layer_id = str(layer_query.layer_id)
|
||||
params = LayerRetrievalParams(
|
||||
rag_session_id=rag_session_id,
|
||||
query_text=c1_query if layer_id == "C1_SYMBOL_CATALOG" and c1_query else effective_query,
|
||||
layer_id=layer_id,
|
||||
limit=max(1, int(layer_query.top_k)),
|
||||
scope=effective_scope,
|
||||
exclude_prefixes=exclude_prefixes,
|
||||
exclude_like=exclude_like,
|
||||
prefer_path_prefixes=prefer_path_prefixes,
|
||||
prefer_like_patterns=prefer_like_patterns,
|
||||
dim=dim,
|
||||
include_tests=test_policy == "INCLUDE",
|
||||
retrieval_mode_hint="symbol_search" if layer_id == "C1_SYMBOL_CATALOG" and c1_query else None,
|
||||
c1_kind_filter=c1_kind_filter if layer_id == "C1_SYMBOL_CATALOG" else None,
|
||||
allow_scope_relaxation=not strict_symbol_mode,
|
||||
)
|
||||
request = {
|
||||
"layer": layer_id,
|
||||
"query": params.query_text,
|
||||
"path_scope": list(path_scope),
|
||||
"include_globs": include_globs,
|
||||
"exclude_globs": exclude_globs,
|
||||
"prefer_globs": prefer_globs,
|
||||
"top_k": params.limit,
|
||||
"ranking_profile": str(getattr(retrieval_spec, "rerank_profile", "") or ""),
|
||||
}
|
||||
report["requests"].append(request)
|
||||
started = perf_counter()
|
||||
rows, layer_diag = self.retrieve_layer_with_scope_fallback(params)
|
||||
report["retrieval_by_layer_ms"][layer_id] = int((perf_counter() - started) * 1000)
|
||||
if test_file_globs:
|
||||
before = len(rows)
|
||||
rows = self._filter_rows_by_globs(rows, test_file_globs)
|
||||
filter_stage = "post_rank"
|
||||
after = len(rows)
|
||||
else:
|
||||
before = len(rows)
|
||||
after = len(rows)
|
||||
filter_stage = "pre_rank"
|
||||
if strict_symbol_mode:
|
||||
rows = self._explain_refiner.refine(layer_id, rows, symbol_candidates)
|
||||
report["applied"].append(
|
||||
{
|
||||
"layer": layer_id,
|
||||
"effective_path_scope": list(layer_diag.get("effective_path_scope") or []),
|
||||
"normalized_include_globs": list(include_prefixes),
|
||||
"normalized_exclude_globs": list(exclude_prefixes or []),
|
||||
"normalized_prefer_globs": list(prefer_path_prefixes),
|
||||
"filter_stage": filter_stage,
|
||||
"candidates_before_filter": before,
|
||||
"candidates_after_filter": after,
|
||||
}
|
||||
)
|
||||
report["executed_layers"].append(layer_id)
|
||||
report["retrieval_mode_by_layer"][layer_id] = str(
|
||||
params.retrieval_mode_hint or layer_diag.get("retrieval_mode") or "disabled"
|
||||
)
|
||||
report["top_k_by_layer"][layer_id] = params.limit
|
||||
report["filters_by_layer"][layer_id] = {
|
||||
"path_scope": list(layer_diag.get("effective_path_scope") or []),
|
||||
"include_globs": list(include_prefixes),
|
||||
"exclude_globs": list(exclude_prefixes or []),
|
||||
"prefer_globs": list(prefer_path_prefixes),
|
||||
}
|
||||
if bool(layer_diag.get("fallback_used")) and params.allow_scope_relaxation:
|
||||
report["fallback"]["used"] = True
|
||||
report["fallback"]["reason"] = str(layer_diag.get("fallback_reason") or "scope_relaxed")
|
||||
merged.extend(rows)
|
||||
self._last_report = report
|
||||
return self._deduper.dedupe(merged)
|
||||
|
||||
def consume_report(self) -> dict[str, Any] | None:
|
||||
report = self._last_report
|
||||
self._last_report = None
|
||||
return report
|
||||
|
||||
def retrieve_layer_with_scope_fallback(self, params: LayerRetrievalParams) -> tuple[list[dict], dict[str, Any]]:
|
||||
options = self._prefixes_options(params.scope) if params.allow_scope_relaxation else [params.scope or None]
|
||||
last_mode = "disabled"
|
||||
for idx, prefixes in enumerate(options):
|
||||
rows, base_mode = self._retrieve_layer(params, prefixes)
|
||||
last_mode = base_mode
|
||||
if params.c1_kind_filter and params.layer_id == "C1_SYMBOL_CATALOG":
|
||||
rows = self._filter_c1_by_kind(rows, params.c1_kind_filter)
|
||||
if rows:
|
||||
retrieval_mode = self._resolve_mode(base_mode, prefixes)
|
||||
fallback_used = idx > 0
|
||||
fallback_reason = "scope_relaxed" if fallback_used else None
|
||||
return rows, {
|
||||
"effective_path_scope": list(prefixes or []),
|
||||
"retrieval_mode": retrieval_mode,
|
||||
"fallback_used": fallback_used,
|
||||
"fallback_reason": fallback_reason,
|
||||
}
|
||||
return [], {
|
||||
"effective_path_scope": [],
|
||||
"retrieval_mode": last_mode,
|
||||
"fallback_used": len(options) > 1,
|
||||
"fallback_reason": "scope_relaxed_no_hits" if len(options) > 1 else None,
|
||||
}
|
||||
|
||||
def _retrieve_layer(self, params: LayerRetrievalParams, prefixes: list[str] | None) -> tuple[list[dict], str]:
|
||||
if params.dim <= 0:
|
||||
if params.layer_id != "C0_SOURCE_CHUNKS":
|
||||
return [], "disabled"
|
||||
return self._repository.retrieve_lexical_code(
|
||||
params.rag_session_id,
|
||||
query_text=params.query_text,
|
||||
limit=params.limit,
|
||||
path_prefixes=prefixes,
|
||||
exclude_path_prefixes=params.exclude_prefixes,
|
||||
exclude_like_patterns=params.exclude_like,
|
||||
prefer_path_prefixes=params.prefer_path_prefixes,
|
||||
prefer_like_patterns=params.prefer_like_patterns,
|
||||
prefer_non_tests=not params.include_tests,
|
||||
), "lexical"
|
||||
return self._repository.retrieve(
|
||||
params.rag_session_id,
|
||||
[0.0] * params.dim,
|
||||
query_text=params.query_text,
|
||||
limit=params.limit,
|
||||
layers=[params.layer_id],
|
||||
path_prefixes=prefixes,
|
||||
exclude_path_prefixes=params.exclude_prefixes,
|
||||
exclude_like_patterns=params.exclude_like,
|
||||
prefer_path_prefixes=params.prefer_path_prefixes,
|
||||
prefer_like_patterns=params.prefer_like_patterns,
|
||||
prefer_non_tests=not params.include_tests,
|
||||
), "vector"
|
||||
|
||||
def _test_filters(self, test_policy: str) -> tuple[list[str] | None, list[str] | None]:
|
||||
if test_policy == "INCLUDE":
|
||||
return None, None
|
||||
filters = build_test_filters()
|
||||
return filters.exclude_path_prefixes, filters.exclude_like_patterns
|
||||
|
||||
def _prefixes_options(self, path_scope: list[str]) -> list[list[str] | None]:
|
||||
if not path_scope:
|
||||
return [None]
|
||||
parent_prefixes = [
|
||||
item.rsplit("/", 1)[0]
|
||||
for item in path_scope
|
||||
if "/" in item and not is_file_path(item)
|
||||
]
|
||||
deduped_parents: list[str] = []
|
||||
for item in parent_prefixes:
|
||||
if item and item not in deduped_parents:
|
||||
deduped_parents.append(item)
|
||||
result: list[list[str] | None] = [path_scope]
|
||||
if deduped_parents:
|
||||
result.append(deduped_parents)
|
||||
result.append(None)
|
||||
return result
|
||||
|
||||
def _resolve_mode(self, base_mode: str, prefixes: list[str] | None) -> str:
|
||||
if base_mode == "disabled":
|
||||
return "disabled"
|
||||
if prefixes:
|
||||
return "exact_path_fetch"
|
||||
return base_mode
|
||||
|
||||
def _empty_report(self) -> dict[str, Any]:
|
||||
return {
|
||||
"executed_layers": [],
|
||||
"retrieval_mode_by_layer": {},
|
||||
"top_k_by_layer": {},
|
||||
"filters_by_layer": {},
|
||||
"requests": [],
|
||||
"applied": [],
|
||||
"fallback": {"used": False, "reason": None},
|
||||
"retrieval_by_layer_ms": {},
|
||||
}
|
||||
|
||||
def _filter_rows_by_globs(self, rows: list[dict], globs: list[str]) -> list[dict]:
|
||||
if not globs:
|
||||
return rows
|
||||
patterns = [item.strip().lower() for item in globs if item.strip()]
|
||||
result: list[dict] = []
|
||||
for row in rows:
|
||||
path = str(row.get("path") or "").lower()
|
||||
if any(fnmatch(path, pattern) or fnmatch(path, f"*/{pattern}") for pattern in patterns):
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
def _c1_query(self, symbol_candidates: list[str], query: str) -> str:
|
||||
tokens = [str(item).strip() for item in symbol_candidates if str(item).strip()]
|
||||
if tokens:
|
||||
return " ".join(tokens)
|
||||
fallback = extract_symbol_tokens(query)
|
||||
return " ".join(fallback) if fallback else str(query or "")
|
||||
|
||||
def _c1_kind_filter(self, symbol_candidates: list[str], symbol_kind_hint: str) -> set[str] | None:
|
||||
if symbol_kind_hint in _DEFAULT_SYMBOL_KINDS or symbol_kind_hint == "module":
|
||||
return {symbol_kind_hint}
|
||||
if symbol_candidates:
|
||||
return set(_DEFAULT_SYMBOL_KINDS)
|
||||
return None
|
||||
|
||||
def _filter_c1_by_kind(self, rows: list[dict], allowed_kinds: set[str]) -> list[dict]:
|
||||
result: list[dict] = []
|
||||
for row in rows:
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
kind = str(metadata.get("kind") or "").strip().lower()
|
||||
if kind in allowed_kinds:
|
||||
result.append(row)
|
||||
return result
|
||||
|
||||
|
||||
class ExplainStrictRefiner:
|
||||
def refine(self, layer_id: str, rows: list[dict], symbol_candidates: list[str]) -> list[dict]:
|
||||
if not rows or not symbol_candidates:
|
||||
return rows
|
||||
sorted_rows = self._sort_exact_first(rows, symbol_candidates)
|
||||
if layer_id != "C4_SEMANTIC_ROLES":
|
||||
return sorted_rows
|
||||
exact = [row for row in sorted_rows if self._is_exact_symbol(row, symbol_candidates)]
|
||||
if not exact:
|
||||
return sorted_rows[:1]
|
||||
primary = exact[:1]
|
||||
same_path = [
|
||||
row
|
||||
for row in sorted_rows
|
||||
if row not in primary and str(row.get("path") or "") == str(primary[0].get("path") or "")
|
||||
]
|
||||
return [*primary, *same_path[:2]]
|
||||
|
||||
def _sort_exact_first(self, rows: list[dict], symbol_candidates: list[str]) -> list[dict]:
|
||||
return sorted(rows, key=lambda row: (0 if self._is_exact_symbol(row, symbol_candidates) else 1))
|
||||
|
||||
def _is_exact_symbol(self, row: dict, symbol_candidates: list[str]) -> bool:
|
||||
title = str(row.get("title") or "").strip()
|
||||
metadata = dict(row.get("metadata") or {})
|
||||
qname = str(metadata.get("qname") or "").strip()
|
||||
symbol_name = str(metadata.get("symbol_name") or "").strip()
|
||||
return any(candidate in {title, qname, symbol_name} for candidate in symbol_candidates)
|
||||
|
||||
|
||||
class ResolvedSymbolSourceHydrator:
|
||||
def __init__(
|
||||
self,
|
||||
retriever: LayerBudgetRetriever,
|
||||
dim_resolver: SessionEmbeddingDimensions,
|
||||
deduper: RagRowDeduplicator,
|
||||
) -> None:
|
||||
self._retriever = retriever
|
||||
self._dim_resolver = dim_resolver
|
||||
self._deduper = deduper
|
||||
|
||||
def hydrate(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
base_query: str,
|
||||
rag_rows: list[dict],
|
||||
symbol_resolution: dict,
|
||||
retrieval_spec,
|
||||
retrieval_constraints,
|
||||
mapper: GlobPatternMapper,
|
||||
) -> list[dict]:
|
||||
if str(symbol_resolution.get("status") or "") != "resolved":
|
||||
return self._deduper.dedupe(rag_rows)
|
||||
resolved = str(symbol_resolution.get("resolved_symbol") or "").strip()
|
||||
if not resolved:
|
||||
return self._deduper.dedupe(rag_rows)
|
||||
alternatives = [str(item).strip() for item in symbol_resolution.get("alternatives", []) if str(item).strip()]
|
||||
target_names = [resolved]
|
||||
for name in alternatives:
|
||||
if name not in target_names:
|
||||
target_names.append(name)
|
||||
if len(target_names) >= 3:
|
||||
break
|
||||
filters = retrieval_spec.filters
|
||||
test_policy = str(getattr(filters, "test_policy", "EXCLUDE") or "EXCLUDE")
|
||||
exclude_prefixes = mapper.to_prefixes(list(getattr(retrieval_constraints, "exclude_globs", []) or []))
|
||||
exclude_like = mapper.to_like_patterns(list(getattr(retrieval_constraints, "exclude_globs", []) or []))
|
||||
prefer_path_prefixes = mapper.to_prefixes(list(getattr(retrieval_constraints, "prefer_globs", []) or []))
|
||||
prefer_like_patterns = mapper.to_like_patterns(list(getattr(retrieval_constraints, "prefer_globs", []) or []))
|
||||
dim = self._dim_resolver.resolve(rag_session_id)
|
||||
c0_limit = self._layer_top_k(retrieval_spec, "C0_SOURCE_CHUNKS", fallback=6)
|
||||
extra_rows: list[dict] = []
|
||||
for symbol_row in self._symbol_rows(rag_rows, target_names):
|
||||
symbol_path = str(symbol_row.get("path") or "").strip()
|
||||
if not symbol_path:
|
||||
continue
|
||||
symbol_meta = dict(symbol_row.get("metadata") or {})
|
||||
symbol_blob_sha = str(symbol_meta.get("blob_sha") or "").strip()
|
||||
params = LayerRetrievalParams(
|
||||
rag_session_id=rag_session_id,
|
||||
query_text=str(symbol_row.get("title") or "").strip() or base_query,
|
||||
layer_id="C0_SOURCE_CHUNKS",
|
||||
limit=min(c0_limit, 4),
|
||||
scope=[symbol_path],
|
||||
exclude_prefixes=exclude_prefixes or None,
|
||||
exclude_like=exclude_like or None,
|
||||
prefer_path_prefixes=prefer_path_prefixes,
|
||||
prefer_like_patterns=prefer_like_patterns,
|
||||
dim=dim,
|
||||
include_tests=test_policy == "INCLUDE",
|
||||
)
|
||||
rows, _ = self._retriever.retrieve_layer_with_scope_fallback(params)
|
||||
for row in rows:
|
||||
row_meta = dict(row.get("metadata") or {})
|
||||
if symbol_blob_sha and str(row_meta.get("blob_sha") or "") != symbol_blob_sha:
|
||||
continue
|
||||
if not self._overlaps_symbol(row, symbol_row.get("span_start"), symbol_row.get("span_end")):
|
||||
continue
|
||||
extra_rows.append(row)
|
||||
return self._deduper.dedupe([*rag_rows, *extra_rows])
|
||||
|
||||
def _symbol_rows(self, rows: list[dict], target_names: list[str]) -> list[dict]:
|
||||
c1_rows = [row for row in rows if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG"]
|
||||
by_id = {
|
||||
str(dict(row.get("metadata") or {}).get("symbol_id") or ""): row
|
||||
for row in c1_rows
|
||||
if str(dict(row.get("metadata") or {}).get("symbol_id") or "").strip()
|
||||
}
|
||||
selected: list[dict] = []
|
||||
seen_ids: set[str] = set()
|
||||
for row in c1_rows:
|
||||
title = str(row.get("title") or "").strip()
|
||||
qname = str(dict(row.get("metadata") or {}).get("qname") or "").strip()
|
||||
if title not in target_names and qname not in target_names:
|
||||
continue
|
||||
symbol_id = str(dict(row.get("metadata") or {}).get("symbol_id") or "")
|
||||
if symbol_id in seen_ids:
|
||||
continue
|
||||
seen_ids.add(symbol_id)
|
||||
selected.append(row)
|
||||
parent_id = str(dict(row.get("metadata") or {}).get("parent_symbol_id") or "")
|
||||
parent = by_id.get(parent_id)
|
||||
if parent is not None and parent_id not in seen_ids:
|
||||
seen_ids.add(parent_id)
|
||||
selected.append(parent)
|
||||
return selected
|
||||
|
||||
def _layer_top_k(self, retrieval_spec, layer_id: str, *, fallback: int) -> int:
|
||||
for item in list(getattr(retrieval_spec, "layer_queries", []) or []):
|
||||
if str(item.layer_id) == layer_id:
|
||||
return max(1, int(item.top_k))
|
||||
return fallback
|
||||
|
||||
def _overlaps_symbol(self, row: dict, symbol_start, symbol_end) -> bool:
|
||||
if symbol_start is None and symbol_end is None:
|
||||
return True
|
||||
row_start = row.get("span_start")
|
||||
row_end = row.get("span_end")
|
||||
if row_start is None or row_end is None:
|
||||
return True
|
||||
start = int(symbol_start if symbol_start is not None else row_start)
|
||||
end = int(symbol_end if symbol_end is not None else row_end)
|
||||
return int(row_start) <= end and int(row_end) >= start
|
||||
|
||||
|
||||
class RagDbAdapter:
|
||||
def __init__(self, repository: RagRepository, dim_resolver: SessionEmbeddingDimensions) -> None:
|
||||
self._repository = repository
|
||||
self._mapper = GlobPatternMapper()
|
||||
self._deduper = RagRowDeduplicator()
|
||||
self._retriever = LayerBudgetRetriever(repository, dim_resolver, self._deduper)
|
||||
self._hydrator = ResolvedSymbolSourceHydrator(self._retriever, dim_resolver, self._deduper)
|
||||
self._last_retrieval_report: dict[str, Any] | None = None
|
||||
|
||||
def retrieve(self, rag_session_id: str, query: str, retrieval_spec, retrieval_constraints=None) -> list[dict]:
|
||||
rows = self._retriever.retrieve(
|
||||
query=query,
|
||||
retrieval_spec=retrieval_spec,
|
||||
retrieval_constraints=retrieval_constraints,
|
||||
mapper=self._mapper,
|
||||
rag_session_id=rag_session_id,
|
||||
query_plan=None,
|
||||
)
|
||||
self._last_retrieval_report = self._retriever.consume_report()
|
||||
return rows
|
||||
|
||||
def retrieve_with_plan(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
query: str,
|
||||
retrieval_spec,
|
||||
retrieval_constraints=None,
|
||||
*,
|
||||
query_plan=None,
|
||||
) -> list[dict]:
|
||||
rows = self._retriever.retrieve(
|
||||
query=query,
|
||||
retrieval_spec=retrieval_spec,
|
||||
retrieval_constraints=retrieval_constraints,
|
||||
mapper=self._mapper,
|
||||
rag_session_id=rag_session_id,
|
||||
query_plan=query_plan,
|
||||
)
|
||||
self._last_retrieval_report = self._retriever.consume_report()
|
||||
return rows
|
||||
|
||||
def hydrate_resolved_symbol_sources(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
base_query: str,
|
||||
rag_rows: list[dict],
|
||||
symbol_resolution: dict,
|
||||
retrieval_spec,
|
||||
retrieval_constraints=None,
|
||||
) -> list[dict]:
|
||||
return self._hydrator.hydrate(
|
||||
rag_session_id=rag_session_id,
|
||||
base_query=base_query,
|
||||
rag_rows=rag_rows,
|
||||
symbol_resolution=symbol_resolution,
|
||||
retrieval_spec=retrieval_spec,
|
||||
retrieval_constraints=retrieval_constraints,
|
||||
mapper=self._mapper,
|
||||
)
|
||||
|
||||
def consume_retrieval_report(self) -> dict[str, Any] | None:
|
||||
report = self._last_retrieval_report
|
||||
self._last_retrieval_report = None
|
||||
return report
|
||||
|
||||
def retrieve_exact_files(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
*,
|
||||
repo_id: str | None = None,
|
||||
paths: list[str],
|
||||
layers: list[str] | None = None,
|
||||
limit: int = 200,
|
||||
query: str = "",
|
||||
ranking_profile: str = "",
|
||||
) -> list[dict]:
|
||||
normalized_paths = normalize_paths(paths)
|
||||
started = perf_counter()
|
||||
rows = self._repository.retrieve_exact_files(
|
||||
rag_session_id,
|
||||
repo_id=repo_id,
|
||||
paths=normalized_paths,
|
||||
layers=layers,
|
||||
limit=limit,
|
||||
)
|
||||
elapsed = int((perf_counter() - started) * 1000)
|
||||
layer_id = list(layers or ["C0_SOURCE_CHUNKS"])[0]
|
||||
self._last_retrieval_report = {
|
||||
"executed_layers": [layer_id],
|
||||
"retrieval_mode_by_layer": {layer_id: "exact_path_fetch"},
|
||||
"top_k_by_layer": {layer_id: int(limit)},
|
||||
"filters_by_layer": {
|
||||
layer_id: {
|
||||
"path_scope": list(normalized_paths),
|
||||
"include_globs": [],
|
||||
"exclude_globs": [],
|
||||
"prefer_globs": [],
|
||||
}
|
||||
},
|
||||
"requests": [
|
||||
{
|
||||
"layer": layer_id,
|
||||
"query": query,
|
||||
"path_scope": list(normalized_paths),
|
||||
"include_globs": [],
|
||||
"exclude_globs": [],
|
||||
"prefer_globs": [],
|
||||
"top_k": int(limit),
|
||||
"ranking_profile": ranking_profile,
|
||||
}
|
||||
],
|
||||
"applied": [
|
||||
{
|
||||
"layer": layer_id,
|
||||
"effective_path_scope": list(normalized_paths),
|
||||
"normalized_include_globs": [],
|
||||
"normalized_exclude_globs": [],
|
||||
"normalized_prefer_globs": [],
|
||||
"filter_stage": "pre_rank",
|
||||
"candidates_before_filter": len(rows),
|
||||
"candidates_after_filter": len(rows),
|
||||
}
|
||||
],
|
||||
"fallback": {"used": False, "reason": None},
|
||||
"retrieval_by_layer_ms": {layer_id: elapsed},
|
||||
}
|
||||
return self._deduper.dedupe(rows)
|
||||
|
||||
def force_symbol_context_c0(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
*,
|
||||
rag_rows: list[dict],
|
||||
symbol_resolution: dict,
|
||||
limit: int = 20,
|
||||
) -> list[dict]:
|
||||
if str(symbol_resolution.get("status") or "") != "resolved":
|
||||
return rag_rows
|
||||
if float(symbol_resolution.get("confidence") or 0.0) < 0.8:
|
||||
return rag_rows
|
||||
target = str(symbol_resolution.get("resolved_symbol") or "").strip()
|
||||
if not target:
|
||||
return rag_rows
|
||||
c1_rows = [row for row in rag_rows if str(row.get("layer") or "") == "C1_SYMBOL_CATALOG"]
|
||||
target_row = next(
|
||||
(
|
||||
row
|
||||
for row in c1_rows
|
||||
if str(row.get("title") or "").strip() == target
|
||||
or str(dict(row.get("metadata") or {}).get("qname") or "").strip() == target
|
||||
),
|
||||
None,
|
||||
)
|
||||
if target_row is None:
|
||||
return rag_rows
|
||||
target_path = normalize_path(str(target_row.get("path") or ""))
|
||||
if not target_path:
|
||||
return rag_rows
|
||||
c0_rows = self.retrieve_exact_files(
|
||||
rag_session_id,
|
||||
paths=[target_path],
|
||||
layers=["C0_SOURCE_CHUNKS"],
|
||||
limit=limit,
|
||||
query=target,
|
||||
ranking_profile="symbol_context",
|
||||
)
|
||||
preserved = [row for row in rag_rows if str(row.get("layer") or "") != "C0_SOURCE_CHUNKS"]
|
||||
return self._deduper.dedupe([*preserved, *c0_rows])
|
||||
@@ -0,0 +1,11 @@
|
||||
from tests.pipeline_setup.utils.rag_indexer import (
|
||||
DeterministicEmbedder,
|
||||
LocalRepoFileCollector,
|
||||
RagSessionIndexer,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"DeterministicEmbedder",
|
||||
"LocalRepoFileCollector",
|
||||
"RagSessionIndexer",
|
||||
]
|
||||
@@ -0,0 +1,127 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import deepcopy
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PipelineResult
|
||||
|
||||
|
||||
class PipelineDiagnosticsBuilder:
|
||||
def build(self, result: PipelineResult) -> dict:
|
||||
payload = deepcopy(result.diagnostics or {})
|
||||
router_plan = dict(payload.get("router_plan") or {})
|
||||
execution = dict(payload.get("execution") or {})
|
||||
retrieval_block = payload.get("retrieval")
|
||||
|
||||
payload["router"] = {
|
||||
"conversation_mode": result.conversation_mode,
|
||||
"keyword_hints": list(router_plan.get("keyword_hints") or []),
|
||||
"path_scope": list(router_plan.get("path_scope") or []),
|
||||
}
|
||||
payload["llm"] = {
|
||||
"used_evidence_count": len(result.rag_rows),
|
||||
"missing_evidence_reason": self._missing_evidence_reason(result),
|
||||
}
|
||||
|
||||
if isinstance(retrieval_block, dict):
|
||||
retrieval_block["symbol_resolution"] = {
|
||||
"status": str(result.symbol_resolution.get("status") or "not_requested"),
|
||||
"resolved_symbol": result.symbol_resolution.get("resolved_symbol"),
|
||||
"alternatives": list(result.symbol_resolution.get("alternatives") or []),
|
||||
}
|
||||
retrieval_block["layers"] = self._layer_diagnostics(result, execution)
|
||||
|
||||
return payload
|
||||
|
||||
def _layer_diagnostics(self, result: PipelineResult, execution: dict) -> list[dict]:
|
||||
layers = list(execution.get("executed_layers") or [])
|
||||
counts = self._layer_hit_counts(result.rag_rows)
|
||||
fallback = {}
|
||||
retrieval = result.diagnostics.get("retrieval")
|
||||
if isinstance(retrieval, dict):
|
||||
fallback = dict(retrieval.get("fallback") or {})
|
||||
reason = fallback.get("reason")
|
||||
|
||||
output: list[dict] = []
|
||||
for layer in layers:
|
||||
hit_count = int(counts.get(str(layer), 0))
|
||||
output.append(
|
||||
{
|
||||
"layer": str(layer),
|
||||
"status": "hit" if hit_count > 0 else "miss",
|
||||
"hit_count": hit_count,
|
||||
"fail_reason": reason if hit_count == 0 else None,
|
||||
}
|
||||
)
|
||||
return output
|
||||
|
||||
def _layer_hit_counts(self, rows: list[dict]) -> dict[str, int]:
|
||||
counts: dict[str, int] = {}
|
||||
for row in rows:
|
||||
layer = str(row.get("layer") or "")
|
||||
counts[layer] = counts.get(layer, 0) + 1
|
||||
return counts
|
||||
|
||||
def _missing_evidence_reason(self, result: PipelineResult) -> str | None:
|
||||
if result.mode != "full_chain":
|
||||
return None
|
||||
answer_policy = dict(result.diagnostics.get("answer_policy") or {})
|
||||
failure_reason = str(answer_policy.get("failure_reason") or "").strip()
|
||||
if failure_reason:
|
||||
return failure_reason
|
||||
if result.rag_rows:
|
||||
return None
|
||||
return "empty_retrieval_context"
|
||||
|
||||
|
||||
class PipelineSummaryBuilder:
|
||||
def build(self, result: PipelineResult, diagnostics: dict) -> dict:
|
||||
router_plan = dict(diagnostics.get("router_plan") or {})
|
||||
retrieval_profile = router_plan.get("retrieval_profile")
|
||||
layers_hit = self._layers_hit(result)
|
||||
evidence_sufficient = len(result.rag_rows) > 0
|
||||
answer_status = self._answer_status(result, evidence_sufficient)
|
||||
|
||||
return {
|
||||
"router": {
|
||||
"intent": result.intent,
|
||||
"sub_intent": router_plan.get("sub_intent"),
|
||||
"confidence": None,
|
||||
},
|
||||
"retrieval": {
|
||||
"profile": retrieval_profile,
|
||||
"layers_hit": layers_hit,
|
||||
"evidence_sufficient": evidence_sufficient,
|
||||
},
|
||||
"llm": {
|
||||
"answer_status": answer_status,
|
||||
"groundedness": self._groundedness(result, evidence_sufficient),
|
||||
},
|
||||
}
|
||||
|
||||
def _layers_hit(self, result: PipelineResult) -> list[str]:
|
||||
layers = {str(row.get("layer") or "") for row in result.rag_rows if str(row.get("layer") or "")}
|
||||
return sorted(layers)
|
||||
|
||||
def _answer_status(self, result: PipelineResult, evidence_sufficient: bool) -> str:
|
||||
if result.mode != "full_chain":
|
||||
return "partial"
|
||||
answer_policy = dict(result.diagnostics.get("answer_policy") or {})
|
||||
policy_mode = str(answer_policy.get("answer_mode") or "").strip()
|
||||
if policy_mode:
|
||||
return policy_mode
|
||||
if not (result.llm_answer or "").strip():
|
||||
return "failed"
|
||||
if not evidence_sufficient:
|
||||
return "insufficient_evidence"
|
||||
return "answered"
|
||||
|
||||
def _groundedness(self, result: PipelineResult, evidence_sufficient: bool) -> str:
|
||||
if result.mode != "full_chain":
|
||||
return "not_applicable"
|
||||
if not (result.llm_answer or "").strip():
|
||||
return "speculative"
|
||||
if evidence_sufficient:
|
||||
return "grounded"
|
||||
return "weakly_grounded"
|
||||
@@ -0,0 +1,135 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from app.modules.agent.code_qa_runtime import CodeQaRuntimeExecutor
|
||||
from app.modules.agent.llm import AgentLlmService
|
||||
from app.modules.agent.prompt_loader import PromptLoader
|
||||
from app.modules.shared.gigachat.client import GigaChatClient
|
||||
from app.modules.shared.gigachat.settings import GigaChatSettings
|
||||
from app.modules.shared.gigachat.token_provider import GigaChatTokenProvider
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.artifact_writer import ArtifactWriter
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.diagnostics import clone_diagnostics
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase, PipelineResult
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.phrases_loader import PhraseCatalogLoader
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_config import PipelineRunConfig
|
||||
|
||||
|
||||
class PipelineRuntime:
|
||||
def __init__(self, mode: str, test_name: str, test_root: Path) -> None:
|
||||
self._config = PipelineRunConfig.from_env(mode=mode, test_name=test_name, test_root=test_root)
|
||||
self._started_at = datetime.now()
|
||||
self._writer = ArtifactWriter(self._config.test_results_dir, test_name=self._config.test_name, run_started_at=self._started_at)
|
||||
self._rag_adapter = None
|
||||
self._session_resolver = None
|
||||
self._executor: CodeQaRuntimeExecutor | None = None
|
||||
|
||||
@property
|
||||
def artifact_path(self) -> Path:
|
||||
return self._writer.path
|
||||
|
||||
def load_cases(self) -> list[PhraseCase]:
|
||||
loader = PhraseCatalogLoader()
|
||||
cases = loader.filter_by_tag(loader.load(self._config.phrases_path), self._config.tag)
|
||||
if not cases:
|
||||
raise ValueError(f"No cases for tag={self._config.tag} in {self._config.phrases_path}")
|
||||
return cases
|
||||
|
||||
def run_router_only(self, case: PhraseCase) -> PipelineResult:
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_runner import RouterOnlyRunner
|
||||
|
||||
result = RouterOnlyRunner(started_at=self._started_at).run_case(case)
|
||||
self._writer.write_record(result.to_record())
|
||||
return result
|
||||
|
||||
def run_router_rag(self, case: PhraseCase) -> PipelineResult:
|
||||
result = self._run_router_rag_case(case)
|
||||
self._writer.write_record(result.to_record())
|
||||
return result
|
||||
|
||||
def run_full_chain(self, case: PhraseCase) -> PipelineResult:
|
||||
result_payload = self._executor_instance().execute(user_query=case.text, rag_session_id=case.rag_session_id or "")
|
||||
route = dict(result_payload.diagnostics.router_result)
|
||||
retrieval = dict(result_payload.diagnostics.retrieval_request)
|
||||
diagnostics = {
|
||||
"router_plan": {
|
||||
"sub_intent": route.get("sub_intent"),
|
||||
"graph_id": route.get("graph_id"),
|
||||
"path_scope": list(retrieval.get("path_scope") or []),
|
||||
"layers": list(retrieval.get("requested_layers") or []),
|
||||
"symbol_candidates": list(result_payload.router_result.query_plan.symbol_candidates or []) if result_payload.router_result else [],
|
||||
},
|
||||
"timings_ms": dict(result_payload.diagnostics.timings_ms or {}),
|
||||
"answer_policy": {
|
||||
"short_circuit": not result_payload.llm_used,
|
||||
"answer_mode": _answer_status(result_payload.answer_mode, result_payload.llm_used),
|
||||
"failure_reason": ",".join(result_payload.validation.reasons),
|
||||
},
|
||||
}
|
||||
result = PipelineResult(
|
||||
case=case,
|
||||
mode="full_chain",
|
||||
run_started_at=self._started_at,
|
||||
rag_session_id=case.rag_session_id,
|
||||
intent=str(route.get("intent") or ""),
|
||||
graph_id=str(route.get("graph_id") or ""),
|
||||
conversation_mode=str(route.get("conversation_mode") or ""),
|
||||
query=str(retrieval.get("query") or case.text),
|
||||
rag_rows=list(result_payload.retrieval_result.raw_rows) if result_payload.retrieval_result else [],
|
||||
symbol_resolution=result_payload.router_result.symbol_resolution.model_dump() if result_payload.router_result else {},
|
||||
llm_answer=result_payload.final_answer,
|
||||
diagnostics=diagnostics,
|
||||
)
|
||||
self._writer.write_record(result.to_record())
|
||||
return result
|
||||
|
||||
def _run_router_rag_case(self, case: PhraseCase) -> PipelineResult:
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_runner import IntentRouterRagPipelineRunner
|
||||
|
||||
rag_adapter, session_resolver = self._rag_components()
|
||||
runner = IntentRouterRagPipelineRunner(
|
||||
started_at=self._started_at,
|
||||
rag_adapter=rag_adapter,
|
||||
session_resolver=session_resolver,
|
||||
)
|
||||
return runner.run_case(case)
|
||||
|
||||
def _rag_components(self):
|
||||
if self._rag_adapter is not None and self._session_resolver is not None:
|
||||
return self._rag_adapter, self._session_resolver
|
||||
from app.modules.rag.persistence.repository import RagRepository
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.rag_db_adapter import RagDbAdapter, SessionEmbeddingDimensions
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.session_resolver import RagSessionResolver
|
||||
|
||||
repository = RagRepository()
|
||||
self._rag_adapter = RagDbAdapter(repository=repository, dim_resolver=SessionEmbeddingDimensions())
|
||||
self._session_resolver = RagSessionResolver(config=self._config, repository=repository)
|
||||
return self._rag_adapter, self._session_resolver
|
||||
|
||||
def _executor_instance(self) -> CodeQaRuntimeExecutor:
|
||||
if self._executor is None:
|
||||
self._executor = CodeQaRuntimeExecutor(_build_llm())
|
||||
return self._executor
|
||||
|
||||
|
||||
def _ms(started: float) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _sub_intent_from_result(result: PipelineResult) -> str | None:
|
||||
router_plan = dict(result.diagnostics.get("router_plan") or {})
|
||||
value = str(router_plan.get("sub_intent") or "").strip()
|
||||
return value or None
|
||||
|
||||
|
||||
def _answer_status(answer_mode: str, llm_used: bool) -> str:
|
||||
if answer_mode == "normal" and llm_used:
|
||||
return "answered"
|
||||
return answer_mode
|
||||
|
||||
|
||||
def _build_llm() -> AgentLlmService:
|
||||
settings = GigaChatSettings.from_env()
|
||||
client = GigaChatClient(settings, GigaChatTokenProvider(settings))
|
||||
return AgentLlmService(client=client, prompts=PromptLoader())
|
||||
@@ -0,0 +1,37 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.modules.rag.persistence.repository import RagRepository
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_config import PipelineRunConfig
|
||||
from tests.pipeline_setup.utils.rag_indexer import RagSessionIndexer
|
||||
|
||||
|
||||
class RagSessionResolver:
|
||||
def __init__(self, config: PipelineRunConfig, repository: RagRepository) -> None:
|
||||
self._config = config
|
||||
self._repository = repository
|
||||
self._indexed_session_id: str | None = None
|
||||
|
||||
def resolve(self, case: PhraseCase) -> str:
|
||||
if self._config.forced_rag_session_id:
|
||||
return self._config.forced_rag_session_id
|
||||
if self._config.reindex_repo_path:
|
||||
return self._resolve_from_reindex(self._config.reindex_repo_path)
|
||||
if case.rag_session_id:
|
||||
return case.rag_session_id
|
||||
if self._config.default_rag_session_id:
|
||||
return self._config.default_rag_session_id
|
||||
raise ValueError(
|
||||
f"Case '{case.case_id}' has no rag_session_id and INTENT_PIPELINE_RAG_SESSION_ID is not set"
|
||||
)
|
||||
|
||||
def _resolve_from_reindex(self, repo_path: Path) -> str:
|
||||
if self._indexed_session_id:
|
||||
return self._indexed_session_id
|
||||
self._indexed_session_id = RagSessionIndexer(self._repository).index_repo(
|
||||
repo_path=repo_path,
|
||||
project_id=self._config.reindex_project_id,
|
||||
)
|
||||
return self._indexed_session_id
|
||||
@@ -0,0 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.answer_quality import AnswerQualityController
|
||||
|
||||
|
||||
def test_quality_controller_removes_meta_and_empty_candidate_sections() -> None:
|
||||
controller = AnswerQualityController()
|
||||
|
||||
answer = (
|
||||
"В файле видны импорты из различных подпакетов. "
|
||||
"Ответ основан исключительно на содержимом указанного файла. "
|
||||
"Кандидаты на health-endpoint: - Нет явных неподтвержденных кандидатов."
|
||||
)
|
||||
|
||||
cleaned = controller.refine(answer, sub_intent="FIND_ENTRYPOINTS")
|
||||
|
||||
assert "различных подпакетов" not in cleaned
|
||||
assert "Ответ основан исключительно" not in cleaned
|
||||
assert "Нет явных неподтвержденных кандидатов" not in cleaned
|
||||
|
||||
|
||||
def test_quality_controller_trims_speculation_after_not_found() -> None:
|
||||
controller = AnswerQualityController()
|
||||
|
||||
answer = (
|
||||
"Сущность RuntimeFactoryManager не найдена в доступном коде. "
|
||||
"Исходя из названия, вероятно этот класс управляет фабриками runtime."
|
||||
)
|
||||
|
||||
cleaned = controller.refine(answer, sub_intent="EXPLAIN")
|
||||
|
||||
assert cleaned == "Сущность RuntimeFactoryManager не найдена в доступном коде."
|
||||
@@ -0,0 +1,202 @@
|
||||
"""Tests for the canonical CODE_QA pipeline (IntentRouterV2 -> retrieval -> evidence -> diagnostics)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.modules.rag.code_qa_pipeline import (
|
||||
CodeChunkItem,
|
||||
CodeQAPipelineRunner,
|
||||
EvidenceBundle,
|
||||
RetrievalRequest,
|
||||
RetrievalResult,
|
||||
)
|
||||
from app.modules.rag.code_qa_pipeline.evidence_gate import evaluate_evidence
|
||||
from app.modules.rag.code_qa_pipeline.retrieval_request_builder import build_retrieval_request
|
||||
from app.modules.rag.code_qa_pipeline.retrieval_result_builder import build_retrieval_result
|
||||
from app.modules.rag.intent_router_v2 import ConversationState, IntentRouterV2
|
||||
from tests.unit_tests.rag.intent_router_testkit import repo_context
|
||||
|
||||
_TEST_ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def _make_router() -> IntentRouterV2:
|
||||
return IntentRouterV2()
|
||||
|
||||
|
||||
def test_router_output_drives_retrieval_request() -> None:
|
||||
"""Router output is directly convertible into RetrievalRequest."""
|
||||
router = _make_router()
|
||||
text = "Открой файл src/mail_order_bot/context.py"
|
||||
result = router.route(text, ConversationState(), repo_context())
|
||||
request = build_retrieval_request(result, "test-session-id")
|
||||
assert request.rag_session_id == "test-session-id"
|
||||
assert request.query
|
||||
assert request.sub_intent == "OPEN_FILE"
|
||||
assert "context.py" in str(request.path_scope) or request.path_scope
|
||||
assert "C0_SOURCE_CHUNKS" in request.requested_layers or request.requested_layers
|
||||
|
||||
|
||||
def test_retrieval_result_normalized_structure() -> None:
|
||||
"""Retrieval returns normalized RetrievalResult with expected fields."""
|
||||
raw = [
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/foo.py", "title": "", "content": "def bar(): pass", "span_start": 1, "span_end": 2, "metadata": {}},
|
||||
{"layer": "C1_SYMBOL_CATALOG", "path": "src/foo.py", "title": "bar", "content": "", "metadata": {"symbol_id": "s1"}},
|
||||
]
|
||||
report = {"executed_layers": ["C0_SOURCE_CHUNKS", "C1_SYMBOL_CATALOG"], "fallback": {"used": False}}
|
||||
sym = {"status": "resolved", "resolved_symbol": "bar"}
|
||||
result = build_retrieval_result(raw, report, sym)
|
||||
assert len(result.code_chunks) == 2
|
||||
assert result.resolved_symbol == "bar"
|
||||
assert result.symbol_resolution_status == "resolved"
|
||||
assert any(c.layer == "C1_SYMBOL_CATALOG" for c in result.code_chunks)
|
||||
assert len(result.layer_outcomes) == 2
|
||||
|
||||
|
||||
def test_evidence_gate_open_file_sufficient() -> None:
|
||||
"""OPEN_FILE with resolved path and chunks passes evidence gate."""
|
||||
chunk = CodeChunkItem(
|
||||
layer="C0_SOURCE_CHUNKS",
|
||||
path="src/context.py",
|
||||
title="",
|
||||
content="x",
|
||||
start_line=1,
|
||||
end_line=2,
|
||||
metadata={},
|
||||
)
|
||||
bundle = EvidenceBundle(
|
||||
resolved_sub_intent="OPEN_FILE",
|
||||
resolved_target="src/context.py",
|
||||
file_candidates=["src/context.py"],
|
||||
evidence_count=2,
|
||||
code_chunks=[chunk, chunk],
|
||||
)
|
||||
decision = evaluate_evidence(bundle)
|
||||
assert decision.passed is True
|
||||
assert not decision.failure_reasons
|
||||
|
||||
|
||||
def test_evidence_gate_open_file_insufficient() -> None:
|
||||
"""OPEN_FILE with no path and no chunks fails evidence gate."""
|
||||
bundle = EvidenceBundle(
|
||||
resolved_sub_intent="OPEN_FILE",
|
||||
resolved_target=None,
|
||||
file_candidates=[],
|
||||
evidence_count=0,
|
||||
)
|
||||
decision = evaluate_evidence(bundle)
|
||||
assert decision.passed is False
|
||||
assert "path_scope_empty" in decision.failure_reasons or "layer_c0_empty" in decision.failure_reasons
|
||||
assert decision.degraded_message
|
||||
|
||||
|
||||
def test_evidence_gate_explain_insufficient() -> None:
|
||||
"""EXPLAIN with no symbol resolution and few chunks fails."""
|
||||
bundle = EvidenceBundle(
|
||||
resolved_sub_intent="EXPLAIN",
|
||||
resolved_target=None,
|
||||
target_type="symbol",
|
||||
evidence_count=1,
|
||||
)
|
||||
decision = evaluate_evidence(bundle)
|
||||
assert decision.passed is False
|
||||
assert "insufficient_evidence" in decision.failure_reasons or "target_not_resolved" in decision.failure_reasons
|
||||
|
||||
|
||||
def test_evidence_gate_find_entrypoints_insufficient() -> None:
|
||||
"""FIND_ENTRYPOINTS with no entrypoints fails."""
|
||||
bundle = EvidenceBundle(
|
||||
resolved_sub_intent="FIND_ENTRYPOINTS",
|
||||
entrypoints=[],
|
||||
evidence_count=0,
|
||||
)
|
||||
decision = evaluate_evidence(bundle)
|
||||
assert decision.passed is False
|
||||
assert "entrypoints_not_found" in decision.failure_reasons
|
||||
|
||||
|
||||
def test_evidence_gate_find_tests_insufficient() -> None:
|
||||
"""FIND_TESTS with no test candidates fails."""
|
||||
bundle = EvidenceBundle(
|
||||
resolved_sub_intent="FIND_TESTS",
|
||||
resolved_target="Context",
|
||||
target_symbol_candidates=["Context"],
|
||||
test_evidence=[],
|
||||
evidence_count=1,
|
||||
)
|
||||
decision = evaluate_evidence(bundle)
|
||||
assert decision.passed is False
|
||||
assert "tests_not_found" in decision.failure_reasons
|
||||
|
||||
|
||||
def test_diagnostics_report_has_failure_reasons() -> None:
|
||||
"""Diagnostics report includes machine-readable failure reasons."""
|
||||
from app.modules.rag.code_qa_pipeline.diagnostics import build_diagnostics_report
|
||||
from app.modules.rag.code_qa_pipeline.evidence_bundle_builder import build_evidence_bundle
|
||||
|
||||
router = _make_router()
|
||||
text = "Где тесты для НесуществующийКласс?"
|
||||
result = router.route(text, ConversationState(), repo_context())
|
||||
request = build_retrieval_request(result, "test-session")
|
||||
retrieval_result = build_retrieval_result([], {}, {"status": "not_found"})
|
||||
bundle = build_evidence_bundle(retrieval_result, result)
|
||||
evaluate_evidence(bundle)
|
||||
report = build_diagnostics_report(
|
||||
router_result=result,
|
||||
retrieval_request=request,
|
||||
retrieval_result=retrieval_result,
|
||||
evidence_bundle=bundle,
|
||||
answer_mode="degraded",
|
||||
)
|
||||
assert report.answer_mode in ("normal", "degraded", "insufficient")
|
||||
assert report.per_layer_outcome is not None
|
||||
assert report.router_result
|
||||
assert report.retrieval_request
|
||||
|
||||
|
||||
def test_canonical_pipeline_with_fake_adapter() -> None:
|
||||
"""Canonical pipeline runs end-to-end with fake adapter; diagnostics and evidence gate applied."""
|
||||
|
||||
class FakeAdapter:
|
||||
def __init__(self, rows: list[dict]) -> None:
|
||||
self._rows = rows
|
||||
self._report: dict | None = None
|
||||
|
||||
def retrieve_with_plan(self, rag_session_id: str, query: str, retrieval_spec, retrieval_constraints=None, *, query_plan=None) -> list[dict]:
|
||||
layers = [str(q.layer_id) for q in (retrieval_spec.layer_queries or [])]
|
||||
self._report = {"executed_layers": layers, "fallback": {"used": False}, "retrieval_by_layer_ms": {}}
|
||||
return list(self._rows)
|
||||
|
||||
def retrieve_exact_files(self, rag_session_id: str, *, repo_id=None, paths: list, layers=None, limit=200, query="", ranking_profile="") -> list[dict]:
|
||||
self._report = {"executed_layers": list(layers or ["C0_SOURCE_CHUNKS"]), "fallback": {"used": False}}
|
||||
return [r for r in self._rows if r.get("path") in paths]
|
||||
|
||||
def hydrate_resolved_symbol_sources(self, rag_session_id: str, base_query: str, rag_rows: list, symbol_resolution: dict, retrieval_spec, retrieval_constraints=None) -> list[dict]:
|
||||
return list(rag_rows)
|
||||
|
||||
def force_symbol_context_c0(self, rag_session_id: str, *, rag_rows: list, symbol_resolution: dict, limit=20) -> list[dict]:
|
||||
return list(rag_rows)
|
||||
|
||||
def consume_retrieval_report(self) -> dict | None:
|
||||
r, self._report = self._report, None
|
||||
return r
|
||||
|
||||
two_chunks = [
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/context.py", "content": "class Context:", "span_start": 1, "span_end": 2, "metadata": {}},
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/context.py", "content": "def run(self): pass", "span_start": 3, "span_end": 4, "metadata": {}},
|
||||
]
|
||||
adapter = FakeAdapter(two_chunks)
|
||||
router = _make_router()
|
||||
runner = CodeQAPipelineRunner(router=router, retrieval_adapter=adapter, repo_context=repo_context())
|
||||
result = runner.run("Открой файл src/context.py", "test-session", run_retrieval=True)
|
||||
assert result.router_result is not None
|
||||
assert result.retrieval_request is not None
|
||||
assert result.retrieval_result is not None
|
||||
assert result.evidence_bundle is not None
|
||||
assert result.diagnostics_report is not None
|
||||
assert result.answer_mode in ("normal", "degraded", "insufficient")
|
||||
assert result.retrieval_result.code_chunks
|
||||
assert result.diagnostics_report.router_result
|
||||
assert result.diagnostics_report.retrieval_request
|
||||
@@ -0,0 +1,65 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.cli import ReindexCliArgs, RunCliArgs, _parse_args
|
||||
|
||||
|
||||
def test_cli_parse_run_defaults_with_legacy_syntax() -> None:
|
||||
parsed = _parse_args(["--mode", "router_only"])
|
||||
assert parsed.command == "run"
|
||||
assert isinstance(parsed.payload, RunCliArgs)
|
||||
args = parsed.payload
|
||||
|
||||
assert args.mode == "router_only"
|
||||
assert args.case_ids == ()
|
||||
assert args.test_name == "cli_router_only"
|
||||
assert args.verbose is False
|
||||
|
||||
|
||||
def test_cli_parse_run_with_overrides() -> None:
|
||||
parsed = _parse_args(
|
||||
[
|
||||
"run",
|
||||
"--mode",
|
||||
"router_rag",
|
||||
"--case-id",
|
||||
"c1",
|
||||
"--case-id",
|
||||
"c2",
|
||||
"--rag-session-id",
|
||||
"sid-1",
|
||||
"--reindex-repo-path",
|
||||
"/tmp/repo",
|
||||
"--reindex-project-id",
|
||||
"proj-1",
|
||||
"--test-name",
|
||||
"run-x",
|
||||
]
|
||||
)
|
||||
assert parsed.command == "run"
|
||||
assert isinstance(parsed.payload, RunCliArgs)
|
||||
args = parsed.payload
|
||||
|
||||
assert args.mode == "router_rag"
|
||||
assert args.case_ids == ("c1", "c2")
|
||||
assert args.rag_session_id == "sid-1"
|
||||
assert args.reindex_repo_path == "/tmp/repo"
|
||||
assert args.reindex_project_id == "proj-1"
|
||||
assert args.test_name == "run-x"
|
||||
assert args.verbose is False
|
||||
|
||||
|
||||
def test_cli_parse_run_verbose_alias() -> None:
|
||||
parsed = _parse_args(["run", "--mode", "router_rag", "--debug"])
|
||||
assert parsed.command == "run"
|
||||
assert isinstance(parsed.payload, RunCliArgs)
|
||||
assert parsed.payload.verbose is True
|
||||
|
||||
|
||||
def test_cli_parse_reindex() -> None:
|
||||
parsed = _parse_args(["reindex", "--repo-path", "/tmp/repo", "--project-id", "proj-1"])
|
||||
assert parsed.command == "reindex"
|
||||
assert isinstance(parsed.payload, ReindexCliArgs)
|
||||
args = parsed.payload
|
||||
|
||||
assert args.repo_path == "/tmp/repo"
|
||||
assert args.project_id == "proj-1"
|
||||
@@ -0,0 +1,368 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
from app.modules.rag.intent_router_v2.models import (
|
||||
CodeRetrievalFilters,
|
||||
EvidencePolicy,
|
||||
IntentRouterResult,
|
||||
LayerQuery,
|
||||
QueryPlan,
|
||||
RetrievalConstraints,
|
||||
RetrievalSpec,
|
||||
SymbolResolution,
|
||||
)
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_runner import IntentRouterRagPipelineRunner, RouterOnlyRunner
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _StaticRouter:
|
||||
result: IntentRouterResult
|
||||
|
||||
def route(self, *_args, **_kwargs) -> IntentRouterResult:
|
||||
return self.result
|
||||
|
||||
|
||||
class _StaticSessionResolver:
|
||||
def resolve(self, _case: PhraseCase) -> str:
|
||||
return "rag-1"
|
||||
|
||||
|
||||
class _FakeRagAdapter:
|
||||
def __init__(self, rows: list[dict]) -> None:
|
||||
self._rows = rows
|
||||
self._last_report: dict | None = None
|
||||
self.exact_calls = 0
|
||||
|
||||
def retrieve(self, rag_session_id: str, query: str, retrieval_spec, retrieval_constraints=None) -> list[dict]:
|
||||
path_scope = list(getattr(retrieval_spec.filters, "path_scope", []) or [])
|
||||
self._last_report = {
|
||||
"executed_layers": [str(item.layer_id) for item in list(retrieval_spec.layer_queries or [])],
|
||||
"retrieval_mode_by_layer": {str(item.layer_id): "exact_path_fetch" for item in retrieval_spec.layer_queries},
|
||||
"top_k_by_layer": {str(item.layer_id): int(item.top_k) for item in retrieval_spec.layer_queries},
|
||||
"filters_by_layer": {
|
||||
str(item.layer_id): {
|
||||
"path_scope": list(path_scope),
|
||||
"include_globs": list(getattr(retrieval_constraints, "include_globs", []) or []),
|
||||
"exclude_globs": list(getattr(retrieval_constraints, "exclude_globs", []) or []),
|
||||
"prefer_globs": list(getattr(retrieval_constraints, "prefer_globs", []) or []),
|
||||
}
|
||||
for item in retrieval_spec.layer_queries
|
||||
},
|
||||
"requests": [
|
||||
{
|
||||
"layer": str(item.layer_id),
|
||||
"query": query,
|
||||
"path_scope": list(path_scope),
|
||||
"include_globs": list(getattr(retrieval_constraints, "include_globs", []) or []),
|
||||
"exclude_globs": list(getattr(retrieval_constraints, "exclude_globs", []) or []),
|
||||
"prefer_globs": list(getattr(retrieval_constraints, "prefer_globs", []) or []),
|
||||
"top_k": int(item.top_k),
|
||||
"ranking_profile": str(getattr(retrieval_spec, "rerank_profile", "") or ""),
|
||||
}
|
||||
for item in retrieval_spec.layer_queries
|
||||
],
|
||||
"applied": [
|
||||
{
|
||||
"layer": str(item.layer_id),
|
||||
"effective_path_scope": list(path_scope),
|
||||
"normalized_include_globs": list(getattr(retrieval_constraints, "include_globs", []) or []),
|
||||
"normalized_exclude_globs": list(getattr(retrieval_constraints, "exclude_globs", []) or []),
|
||||
"normalized_prefer_globs": list(getattr(retrieval_constraints, "prefer_globs", []) or []),
|
||||
"filter_stage": "pre_rank",
|
||||
"candidates_before_filter": len(self._rows),
|
||||
"candidates_after_filter": len(self._rows),
|
||||
}
|
||||
for item in retrieval_spec.layer_queries
|
||||
],
|
||||
"fallback": {"used": False, "reason": None},
|
||||
"retrieval_by_layer_ms": {str(item.layer_id): 1 for item in retrieval_spec.layer_queries},
|
||||
}
|
||||
return list(self._rows)
|
||||
|
||||
def retrieve_with_plan(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
query: str,
|
||||
retrieval_spec,
|
||||
retrieval_constraints=None,
|
||||
*,
|
||||
query_plan=None,
|
||||
) -> list[dict]:
|
||||
return self.retrieve(rag_session_id, query, retrieval_spec, retrieval_constraints)
|
||||
|
||||
def consume_retrieval_report(self) -> dict | None:
|
||||
report = self._last_report
|
||||
self._last_report = None
|
||||
return report
|
||||
|
||||
def retrieve_exact_files(
|
||||
self,
|
||||
*,
|
||||
rag_session_id: str,
|
||||
repo_id: str | None = None,
|
||||
paths: list[str],
|
||||
layers: list[str] | None = None,
|
||||
limit: int = 200,
|
||||
query: str = "",
|
||||
ranking_profile: str = "",
|
||||
) -> list[dict]:
|
||||
self.exact_calls += 1
|
||||
layer = list(layers or ["C0_SOURCE_CHUNKS"])[0]
|
||||
normalized = list(paths)
|
||||
matched = [row for row in self._rows if str(row.get("path") or "") in set(normalized)]
|
||||
self._last_report = {
|
||||
"executed_layers": [layer],
|
||||
"retrieval_mode_by_layer": {layer: "exact_path_fetch"},
|
||||
"top_k_by_layer": {layer: int(limit)},
|
||||
"filters_by_layer": {
|
||||
layer: {
|
||||
"path_scope": normalized,
|
||||
"include_globs": [],
|
||||
"exclude_globs": [],
|
||||
"prefer_globs": [],
|
||||
}
|
||||
},
|
||||
"requests": [
|
||||
{
|
||||
"layer": layer,
|
||||
"query": query,
|
||||
"path_scope": normalized,
|
||||
"include_globs": [],
|
||||
"exclude_globs": [],
|
||||
"prefer_globs": [],
|
||||
"top_k": int(limit),
|
||||
"ranking_profile": ranking_profile,
|
||||
}
|
||||
],
|
||||
"applied": [
|
||||
{
|
||||
"layer": layer,
|
||||
"effective_path_scope": normalized,
|
||||
"normalized_include_globs": [],
|
||||
"normalized_exclude_globs": [],
|
||||
"normalized_prefer_globs": [],
|
||||
"filter_stage": "pre_rank",
|
||||
"candidates_before_filter": len(matched),
|
||||
"candidates_after_filter": len(matched),
|
||||
}
|
||||
],
|
||||
"fallback": {"used": False, "reason": None},
|
||||
"retrieval_by_layer_ms": {layer: 1},
|
||||
}
|
||||
return list(matched)
|
||||
|
||||
def hydrate_resolved_symbol_sources(self, **kwargs) -> list[dict]:
|
||||
return list(kwargs["rag_rows"])
|
||||
|
||||
def force_symbol_context_c0(self, rag_session_id: str, *, rag_rows: list[dict], symbol_resolution: dict, limit: int = 20) -> list[dict]: # noqa: ARG002
|
||||
return list(rag_rows)
|
||||
|
||||
|
||||
def _open_file_result() -> IntentRouterResult:
|
||||
path_scope = ["src/mail_order_bot/context.py"]
|
||||
return IntentRouterResult(
|
||||
intent="CODE_QA",
|
||||
retrieval_profile="code",
|
||||
graph_id="CodeQAGraph",
|
||||
conversation_mode="START",
|
||||
query_plan=QueryPlan(
|
||||
raw="Открой файл src/mail_order_bot/context.py",
|
||||
normalized="Открой файл src/mail_order_bot/context.py",
|
||||
sub_intent="OPEN_FILE",
|
||||
keyword_hints=[],
|
||||
path_hints=list(path_scope),
|
||||
doc_scope_hints=[],
|
||||
symbol_candidates=[],
|
||||
symbol_kind_hint="unknown",
|
||||
),
|
||||
retrieval_spec=RetrievalSpec(
|
||||
domains=["CODE"],
|
||||
layer_queries=[LayerQuery(layer_id="C0_SOURCE_CHUNKS", top_k=8)],
|
||||
filters=CodeRetrievalFilters(test_policy="EXCLUDE", path_scope=list(path_scope), language=[]),
|
||||
rerank_profile="code",
|
||||
),
|
||||
retrieval_constraints=RetrievalConstraints(
|
||||
include_globs=list(path_scope),
|
||||
exclude_globs=["tests/**"],
|
||||
prefer_globs=[],
|
||||
test_file_globs=[],
|
||||
test_symbol_patterns=[],
|
||||
max_candidates=20,
|
||||
),
|
||||
symbol_resolution=SymbolResolution(status="not_requested"),
|
||||
evidence_policy=EvidencePolicy(require_spec=False),
|
||||
)
|
||||
|
||||
|
||||
def _explain_result() -> IntentRouterResult:
|
||||
return IntentRouterResult(
|
||||
intent="CODE_QA",
|
||||
retrieval_profile="code",
|
||||
graph_id="CodeQAGraph",
|
||||
conversation_mode="START",
|
||||
query_plan=QueryPlan(
|
||||
raw="Объясни как работает Context",
|
||||
normalized="Объясни как работает Context",
|
||||
sub_intent="EXPLAIN",
|
||||
keyword_hints=["Context"],
|
||||
path_hints=[],
|
||||
doc_scope_hints=[],
|
||||
symbol_candidates=["Context"],
|
||||
symbol_kind_hint="class",
|
||||
),
|
||||
retrieval_spec=RetrievalSpec(
|
||||
domains=["CODE"],
|
||||
layer_queries=[LayerQuery(layer_id="C1_SYMBOL_CATALOG", top_k=8), LayerQuery(layer_id="C0_SOURCE_CHUNKS", top_k=8)],
|
||||
filters=CodeRetrievalFilters(test_policy="EXCLUDE", path_scope=["src/mail_order_bot/context.py"], language=[]),
|
||||
rerank_profile="code",
|
||||
),
|
||||
retrieval_constraints=RetrievalConstraints(
|
||||
include_globs=["src/**"],
|
||||
exclude_globs=["tests/**"],
|
||||
prefer_globs=[],
|
||||
test_file_globs=[],
|
||||
test_symbol_patterns=[],
|
||||
max_candidates=20,
|
||||
),
|
||||
symbol_resolution=SymbolResolution(status="pending", alternatives=["Context"], confidence=0.0),
|
||||
evidence_policy=EvidencePolicy(require_spec=False),
|
||||
)
|
||||
|
||||
|
||||
def test_router_only_diagnostics_contains_router_plan() -> None:
|
||||
case = PhraseCase(case_id="c1", text="Открой файл src/mail_order_bot/context.py")
|
||||
runner = RouterOnlyRunner(started_at=datetime(2026, 3, 5, 12, 0, 0), router=_StaticRouter(_open_file_result()))
|
||||
|
||||
record = runner.run_case(case).to_record()
|
||||
|
||||
assert record["summary"]["router"]["intent"] == "CODE_QA"
|
||||
assert record["summary"]["llm"]["answer_status"] == "partial"
|
||||
assert record["diagnostics"]["router_plan"]["sub_intent"] == "OPEN_FILE"
|
||||
assert record["diagnostics"]["router"]["conversation_mode"] == "START"
|
||||
assert record["diagnostics"]["llm"]["used_evidence_count"] == 0
|
||||
assert record["diagnostics"]["execution"]["executed_layers"] == []
|
||||
assert record["diagnostics"]["retrieval"] is None
|
||||
assert record["diagnostics"]["timings_ms"]["router"] >= 0
|
||||
|
||||
|
||||
def test_router_rag_diagnostics_keeps_plan_and_request_scope() -> None:
|
||||
case = PhraseCase(case_id="c2", text="Открой файл src/mail_order_bot/context.py")
|
||||
route_result = _open_file_result()
|
||||
rag_adapter = _FakeRagAdapter(
|
||||
rows=[
|
||||
{
|
||||
"path": "src/mail_order_bot/context.py",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/mail_order_bot/context.py:1-10",
|
||||
"content": "class Context: ...",
|
||||
"metadata": {"repo_id": "MailOrderBot"},
|
||||
}
|
||||
]
|
||||
)
|
||||
runner = IntentRouterRagPipelineRunner(
|
||||
started_at=datetime(2026, 3, 5, 12, 0, 0),
|
||||
rag_adapter=rag_adapter,
|
||||
session_resolver=_StaticSessionResolver(),
|
||||
router=_StaticRouter(route_result),
|
||||
)
|
||||
|
||||
record = runner.run_case(case).to_record()
|
||||
diagnostics = record["diagnostics"]
|
||||
|
||||
assert record["summary"]["retrieval"]["layers_hit"] == ["C0_SOURCE_CHUNKS"]
|
||||
assert diagnostics["router_plan"]["path_scope"] == ["src/mail_order_bot/context.py"]
|
||||
assert diagnostics["router_plan"]["retrieval_constraints"]["include_globs"] == ["src/mail_order_bot/context.py"]
|
||||
assert diagnostics["retrieval"]["requests"][0]["path_scope"] == diagnostics["router_plan"]["path_scope"]
|
||||
assert diagnostics["retrieval"]["symbol_resolution"]["status"] == "not_requested"
|
||||
assert diagnostics["retrieval"]["layers"][0]["layer"] == "C0_SOURCE_CHUNKS"
|
||||
assert diagnostics["retrieval"]["fallback"]["used"] is False
|
||||
assert rag_adapter.exact_calls == 1
|
||||
|
||||
|
||||
def test_open_file_strict_exact_path_returns_only_target_and_no_violation() -> None:
|
||||
case = PhraseCase(case_id="c4", text="Открой файл src/mail_order_bot/context.py")
|
||||
rag_adapter = _FakeRagAdapter(
|
||||
rows=[
|
||||
{
|
||||
"path": "src/mail_order_bot/context.py",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/mail_order_bot/context.py:1-10",
|
||||
"content": "class Context: ...",
|
||||
"metadata": {"repo_id": "MailOrderBot"},
|
||||
},
|
||||
{
|
||||
"path": "src/mail_order_bot/other.py",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/mail_order_bot/other.py:1-10",
|
||||
"content": "def helper(): ...",
|
||||
"metadata": {"repo_id": "MailOrderBot"},
|
||||
},
|
||||
]
|
||||
)
|
||||
runner = IntentRouterRagPipelineRunner(
|
||||
started_at=datetime(2026, 3, 5, 12, 0, 0),
|
||||
rag_adapter=rag_adapter,
|
||||
session_resolver=_StaticSessionResolver(),
|
||||
router=_StaticRouter(_open_file_result()),
|
||||
)
|
||||
|
||||
result = runner.run_case(case)
|
||||
record = result.to_record()
|
||||
diagnostics = record["diagnostics"]
|
||||
|
||||
assert result.rag_rows
|
||||
assert all(str(row.get("path") or "") == "src/mail_order_bot/context.py" for row in result.rag_rows)
|
||||
assert diagnostics["execution"]["filters_by_layer"]["C0_SOURCE_CHUNKS"]["path_scope"] == [
|
||||
"src/mail_order_bot/context.py"
|
||||
]
|
||||
assert diagnostics["retrieval"]["applied"][0]["effective_path_scope"] == ["src/mail_order_bot/context.py"]
|
||||
assert diagnostics["retrieval"]["fallback"]["used"] is False
|
||||
assert diagnostics["constraint_violations"] == []
|
||||
|
||||
|
||||
def test_router_rag_open_file_violation_is_reported_when_target_file_not_returned() -> None:
|
||||
case = PhraseCase(case_id="c3", text="Открой файл src/mail_order_bot/context.py")
|
||||
rag_adapter = _FakeRagAdapter(
|
||||
rows=[
|
||||
{
|
||||
"path": "src/mail_order_bot/other.py",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/mail_order_bot/other.py:1-10",
|
||||
"content": "def helper(): ...",
|
||||
"metadata": {"repo_id": "MailOrderBot"},
|
||||
}
|
||||
]
|
||||
)
|
||||
runner = IntentRouterRagPipelineRunner(
|
||||
started_at=datetime(2026, 3, 5, 12, 0, 0),
|
||||
rag_adapter=rag_adapter,
|
||||
session_resolver=_StaticSessionResolver(),
|
||||
router=_StaticRouter(_open_file_result()),
|
||||
)
|
||||
|
||||
record = runner.run_case(case).to_record()
|
||||
violation_types = [item["type"] for item in record["diagnostics"]["constraint_violations"]]
|
||||
|
||||
assert "PATH_SCOPE_NOT_SATISFIED" in violation_types
|
||||
|
||||
|
||||
def test_explain_with_symbol_candidates_returns_empty_rag_when_symbol_not_resolved() -> None:
|
||||
case = PhraseCase(case_id="c5", text="Объясни как работает Context")
|
||||
rag_adapter = _FakeRagAdapter(rows=[])
|
||||
runner = IntentRouterRagPipelineRunner(
|
||||
started_at=datetime(2026, 3, 5, 12, 0, 0),
|
||||
rag_adapter=rag_adapter,
|
||||
session_resolver=_StaticSessionResolver(),
|
||||
router=_StaticRouter(_explain_result()),
|
||||
)
|
||||
|
||||
result = runner.run_case(case)
|
||||
record = result.to_record()
|
||||
violation_types = [item["type"] for item in record["diagnostics"]["constraint_violations"]]
|
||||
|
||||
assert result.rag_rows == []
|
||||
assert record["diagnostics"]["retrieval"]["fallback"]["used"] is False
|
||||
assert "SYMBOL_RESOLUTION_FAILED" in violation_types
|
||||
@@ -0,0 +1,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.runtime import PipelineRuntime
|
||||
|
||||
_TEST_ROOT = Path(__file__).resolve().parent
|
||||
_RUNTIME = PipelineRuntime(mode="router_only", test_name="test_intent_router_only_matrix", test_root=_TEST_ROOT)
|
||||
_CASES = _RUNTIME.load_cases()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", _CASES, ids=lambda item: item.case_id)
|
||||
def test_intent_router_only_matrix(case) -> None:
|
||||
result = _RUNTIME.run_router_only(case)
|
||||
|
||||
assert result.intent in {"CODE_QA", "DOCS_QA", "GENERATE_DOCS_FROM_CODE", "PROJECT_MISC"}
|
||||
if case.expected_intent:
|
||||
assert result.intent == case.expected_intent
|
||||
@@ -0,0 +1,31 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_config import mode_enabled
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.runtime import PipelineRuntime
|
||||
|
||||
pytestmark = pytest.mark.full_chain
|
||||
|
||||
_TEST_ROOT = Path(__file__).resolve().parent
|
||||
_RUNTIME = PipelineRuntime(mode="full_chain", test_name="test_intent_router_rag_llm_pipeline", test_root=_TEST_ROOT)
|
||||
_CASES = _RUNTIME.load_cases()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not mode_enabled("full_chain", _TEST_ROOT), reason="set RUN_INTENT_PIPELINE_FULL_CHAIN=1 to run")
|
||||
@pytest.mark.parametrize("case", _CASES, ids=lambda item: item.case_id)
|
||||
def test_intent_router_rag_llm_pipeline(case) -> None:
|
||||
pytest.importorskip("sqlalchemy", reason="full_chain integration requires sqlalchemy dependency")
|
||||
|
||||
try:
|
||||
result = _RUNTIME.run_full_chain(case)
|
||||
except ValueError as exc:
|
||||
pytest.skip(str(exc))
|
||||
|
||||
if case.expected_intent:
|
||||
assert result.intent == case.expected_intent
|
||||
if case.expect_non_empty_rag:
|
||||
assert result.rag_rows, f"RAG returned empty list for case={case.case_id} rag_session_id={result.rag_session_id}"
|
||||
assert (result.llm_answer or "").strip(), "LLM answer must not be empty"
|
||||
@@ -0,0 +1,27 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.pipeline_config import mode_enabled
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.runtime import PipelineRuntime
|
||||
|
||||
pytestmark = pytest.mark.router_rag
|
||||
|
||||
_TEST_ROOT = Path(__file__).resolve().parent
|
||||
_RUNTIME = PipelineRuntime(mode="router_rag", test_name="test_intent_router_rag_pipeline", test_root=_TEST_ROOT)
|
||||
_CASES = _RUNTIME.load_cases()
|
||||
|
||||
|
||||
@pytest.mark.skipif(not mode_enabled("router_rag", _TEST_ROOT), reason="set RUN_INTENT_PIPELINE_ROUTER_RAG=1 to run")
|
||||
@pytest.mark.parametrize("case", _CASES, ids=lambda item: item.case_id)
|
||||
def test_intent_router_rag_pipeline(case) -> None:
|
||||
pytest.importorskip("sqlalchemy", reason="router_rag integration requires sqlalchemy dependency")
|
||||
|
||||
result = _RUNTIME.run_router_rag(case)
|
||||
|
||||
if case.expected_intent:
|
||||
assert result.intent == case.expected_intent
|
||||
if case.expect_non_empty_rag:
|
||||
assert result.rag_rows, f"RAG returned empty list for case={case.case_id} rag_session_id={result.rag_session_id}"
|
||||
@@ -0,0 +1,80 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.prompt_payload_builder import PromptPayloadBuilder
|
||||
|
||||
|
||||
def test_build_prompt_payload_includes_layer_guide_and_explain_quotas() -> None:
|
||||
builder = PromptPayloadBuilder()
|
||||
rag_rows = [
|
||||
{"layer": "C1_SYMBOL_CATALOG", "path": "src/context.py", "title": "Context", "content": "class Context"},
|
||||
{"layer": "C1_SYMBOL_CATALOG", "path": "src/other.py", "title": "Other", "content": "class Other"},
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/context.py", "title": "Context chunk 1", "content": "def start(self): pass"},
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/context.py", "title": "Context chunk 2", "content": "def stop(self): pass"},
|
||||
{"layer": "C0_SOURCE_CHUNKS", "path": "src/context.py", "title": "Context chunk 3", "content": "def status(self): pass"},
|
||||
{"layer": "C2_DEPENDENCY_GRAPH", "path": "src/context.py", "title": "Context->Worker", "content": "Context calls Worker"},
|
||||
{"layer": "C4_SEMANTIC_ROLES", "path": "src/context.py", "title": "Context", "content": "role: pipeline_stage"},
|
||||
{"layer": "C4_SEMANTIC_ROLES", "path": "src/control.py", "title": "ControlChannel", "content": "role: model"},
|
||||
{"layer": "C3_ENTRYPOINTS", "path": "src/main.py", "title": "main", "content": "entrypoint"},
|
||||
]
|
||||
|
||||
payload = builder.build(
|
||||
"Объясни как работает Context",
|
||||
rag_rows,
|
||||
prompt_template={"template_id": "t", "system_prompt": "sys", "task_prompt": "task"},
|
||||
sub_intent="EXPLAIN",
|
||||
default_system_prompt="sys-default",
|
||||
default_task_prompt="task-default",
|
||||
default_template_id="default",
|
||||
system_prompt_version="v1",
|
||||
)
|
||||
user_prompt = payload["user_prompt"]
|
||||
|
||||
assert "Как интерпретировать слои RAG" in user_prompt
|
||||
assert user_prompt.count("[C1_SYMBOL_CATALOG]") == 1
|
||||
assert user_prompt.count("[C0_SOURCE_CHUNKS]") == 2
|
||||
assert user_prompt.count("[C2_DEPENDENCY_GRAPH]") == 1
|
||||
assert user_prompt.count("[C4_SEMANTIC_ROLES]") == 1
|
||||
assert user_prompt.count("[C3_ENTRYPOINTS]") == 1
|
||||
|
||||
|
||||
def test_find_entrypoints_prompt_prefers_http_route_rows() -> None:
|
||||
builder = PromptPayloadBuilder()
|
||||
rag_rows = [
|
||||
{
|
||||
"layer": "C3_ENTRYPOINTS",
|
||||
"path": "src/app.py",
|
||||
"title": "candidate",
|
||||
"content": "lifecycle start",
|
||||
"metadata": {"entry_type": "startup"},
|
||||
},
|
||||
{
|
||||
"layer": "C3_ENTRYPOINTS",
|
||||
"path": "src/http_app.py",
|
||||
"title": "GET /health declared in HttpControlAppFactory",
|
||||
"content": "raw",
|
||||
"metadata": {
|
||||
"http_method": "GET",
|
||||
"route_path": "/health",
|
||||
"declaring_symbol": "HttpControlAppFactory",
|
||||
"handler_symbol": "HttpControlAppFactory.create.health",
|
||||
"decorator_text": "@app.get('/health')",
|
||||
"summary_text": "GET /health declared in HttpControlAppFactory",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
payload = builder.build(
|
||||
"Где health endpoint?",
|
||||
rag_rows,
|
||||
prompt_template={"template_id": "t", "system_prompt": "sys", "task_prompt": "task"},
|
||||
sub_intent="FIND_ENTRYPOINTS",
|
||||
default_system_prompt="sys-default",
|
||||
default_task_prompt="task-default",
|
||||
default_template_id="default",
|
||||
system_prompt_version="v1",
|
||||
)
|
||||
user_prompt = payload["user_prompt"]
|
||||
|
||||
assert "GET /health declared in HttpControlAppFactory" in user_prompt
|
||||
assert "Declared in: HttpControlAppFactory" in user_prompt
|
||||
assert "Handler: HttpControlAppFactory.create.health" in user_prompt
|
||||
@@ -0,0 +1,49 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.llm_prompt_loader import LlmPromptCatalogLoader
|
||||
|
||||
_PROMPT_CATALOG = Path(__file__).resolve().parents[3] / "pipeline_setup_v2" / "llm_prompts.yaml"
|
||||
|
||||
|
||||
def test_prompt_catalog_loads_intent_templates() -> None:
|
||||
loader = LlmPromptCatalogLoader()
|
||||
catalog = loader.load(_PROMPT_CATALOG)
|
||||
|
||||
code_prompt = loader.select_for_intent(catalog, "CODE_QA")
|
||||
|
||||
assert code_prompt["template_id"] == "intent_code_qa_v5"
|
||||
assert "senior Python-инженер" in code_prompt["system_prompt"]
|
||||
assert "естественным инженерным языком" in code_prompt["task_prompt"]
|
||||
|
||||
|
||||
def test_prompt_catalog_prefers_sub_intent_template() -> None:
|
||||
loader = LlmPromptCatalogLoader()
|
||||
catalog = loader.load(_PROMPT_CATALOG)
|
||||
|
||||
prompt = loader.select(catalog, intent="CODE_QA", sub_intent="FIND_TESTS")
|
||||
|
||||
assert prompt["template_id"] == "intent_code_qa_find_tests_ru_v4"
|
||||
assert "тестовое покрытие" in prompt["system_prompt"]
|
||||
assert "прямых тестов нет" in prompt["task_prompt"]
|
||||
|
||||
|
||||
def test_prompt_catalog_falls_back_to_intent_template_for_unknown_sub_intent() -> None:
|
||||
loader = LlmPromptCatalogLoader()
|
||||
catalog = loader.load(_PROMPT_CATALOG)
|
||||
|
||||
prompt = loader.select(catalog, intent="CODE_QA", sub_intent="UNKNOWN_SUB_INTENT")
|
||||
|
||||
assert prompt["template_id"] == "intent_code_qa_v5"
|
||||
assert "естественным инженерным языком" in prompt["task_prompt"]
|
||||
|
||||
|
||||
def test_prompt_catalog_falls_back_to_default_for_unknown_intent() -> None:
|
||||
loader = LlmPromptCatalogLoader()
|
||||
catalog = loader.load(_PROMPT_CATALOG)
|
||||
|
||||
prompt = loader.select_for_intent(catalog, "UNKNOWN_INTENT")
|
||||
|
||||
assert prompt["template_id"] == "intent_default_v2"
|
||||
assert "технический ассистент" in prompt["system_prompt"]
|
||||
@@ -0,0 +1,25 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.phrases_loader import PhraseCatalogLoader
|
||||
|
||||
|
||||
def test_phrase_catalog_has_cases_and_unique_ids() -> None:
|
||||
path = Path(__file__).resolve().parent / "fixtures" / "phrases.yaml"
|
||||
cases = PhraseCatalogLoader().load(path)
|
||||
|
||||
assert len(cases) >= 5
|
||||
case_ids = [item.case_id for item in cases]
|
||||
assert len(case_ids) == len(set(case_ids))
|
||||
assert all(item.text for item in cases)
|
||||
|
||||
|
||||
def test_phrase_catalog_has_tags_for_all_iterations() -> None:
|
||||
path = Path(__file__).resolve().parent / "fixtures" / "phrases.yaml"
|
||||
loader = PhraseCatalogLoader()
|
||||
cases = loader.load(path)
|
||||
|
||||
assert loader.filter_by_tag(cases, "router_only")
|
||||
assert loader.filter_by_tag(cases, "router_rag")
|
||||
assert loader.filter_by_tag(cases, "full_chain")
|
||||
@@ -0,0 +1,312 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from types import SimpleNamespace
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.rag_db_adapter import RagDbAdapter
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _LayerQuery:
|
||||
layer_id: str
|
||||
top_k: int
|
||||
|
||||
|
||||
class _FakeDimResolver:
|
||||
def resolve(self, rag_session_id: str) -> int: # noqa: ARG002
|
||||
return 4
|
||||
|
||||
|
||||
class _RecordingRepository:
|
||||
def __init__(self) -> None:
|
||||
self.retrieve_calls: list[dict] = []
|
||||
self.retrieve_lexical_calls: list[dict] = []
|
||||
self.retrieve_exact_calls: list[dict] = []
|
||||
|
||||
def retrieve(self, rag_session_id: str, query_embedding: list[float], **kwargs) -> list[dict]: # noqa: ARG002
|
||||
self.retrieve_calls.append(kwargs)
|
||||
layer = list(kwargs.get("layers") or [""])[0]
|
||||
if layer == "C1_SYMBOL_CATALOG":
|
||||
return [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context",
|
||||
"layer": layer,
|
||||
"title": "Context",
|
||||
"span_start": 10,
|
||||
"span_end": 20,
|
||||
"metadata": {"symbol_id": "sym-1", "qname": "Context", "kind": "class"},
|
||||
},
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context duplicate",
|
||||
"layer": layer,
|
||||
"title": "Context",
|
||||
"span_start": 10,
|
||||
"span_end": 20,
|
||||
"metadata": {"symbol_id": "sym-1", "qname": "Context", "kind": "class"},
|
||||
},
|
||||
]
|
||||
if layer == "C0_SOURCE_CHUNKS":
|
||||
return [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context: ...",
|
||||
"layer": layer,
|
||||
"title": "src/context.py:Context",
|
||||
"span_start": 10,
|
||||
"span_end": 30,
|
||||
"metadata": {"chunk_index": 0, "blob_sha": "blob-1"},
|
||||
},
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context: ... duplicate",
|
||||
"layer": layer,
|
||||
"title": "src/context.py:Context",
|
||||
"span_start": 10,
|
||||
"span_end": 30,
|
||||
"metadata": {"chunk_index": 1, "blob_sha": "blob-1"},
|
||||
},
|
||||
]
|
||||
if layer == "C2_DEPENDENCY_GRAPH":
|
||||
return [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "Context.set writes_attr Context.data",
|
||||
"layer": layer,
|
||||
"title": "Context.set:writes_attr",
|
||||
"span_start": 40,
|
||||
"span_end": 40,
|
||||
"metadata": {"edge_id": "edge-1"},
|
||||
}
|
||||
]
|
||||
if layer == "C4_SEMANTIC_ROLES":
|
||||
return [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "Context\nrole: pipeline_stage\n\nResponsibilities:\n- reads and writes state attributes",
|
||||
"layer": layer,
|
||||
"title": "Context",
|
||||
"span_start": 10,
|
||||
"span_end": 30,
|
||||
"lexical_rank": 0,
|
||||
"structural_rank": 10,
|
||||
"metadata": {"symbol_name": "Context", "qname": "Context", "role": "pipeline_stage"},
|
||||
},
|
||||
{
|
||||
"path": "src/control.py",
|
||||
"content": "ControlChannel\nrole: model\n\nResponsibilities:\n- default model role",
|
||||
"layer": layer,
|
||||
"title": "ControlChannel",
|
||||
"span_start": 1,
|
||||
"span_end": 20,
|
||||
"lexical_rank": 3,
|
||||
"structural_rank": 40,
|
||||
"metadata": {"symbol_name": "ControlChannel", "qname": "ControlChannel", "role": "model"},
|
||||
},
|
||||
]
|
||||
return []
|
||||
|
||||
def retrieve_lexical_code(self, rag_session_id: str, **kwargs) -> list[dict]: # noqa: ARG002
|
||||
self.retrieve_lexical_calls.append(kwargs)
|
||||
return []
|
||||
|
||||
def retrieve_exact_files(
|
||||
self,
|
||||
rag_session_id: str,
|
||||
*,
|
||||
repo_id: str | None = None,
|
||||
paths: list[str],
|
||||
layers: list[str] | None = None,
|
||||
limit: int = 200,
|
||||
) -> list[dict]: # noqa: ARG002
|
||||
self.retrieve_exact_calls.append({"paths": list(paths), "layers": list(layers or []), "limit": limit})
|
||||
return [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context: ...",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/context.py:Context",
|
||||
"span_start": 10,
|
||||
"span_end": 30,
|
||||
"metadata": {"chunk_index": 0, "blob_sha": "blob-1"},
|
||||
},
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context: duplicate",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/context.py:Context",
|
||||
"span_start": 10,
|
||||
"span_end": 30,
|
||||
"metadata": {"chunk_index": 1, "blob_sha": "blob-1"},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def _spec() -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
layer_queries=[
|
||||
_LayerQuery(layer_id="C1_SYMBOL_CATALOG", top_k=2),
|
||||
_LayerQuery(layer_id="C0_SOURCE_CHUNKS", top_k=3),
|
||||
_LayerQuery(layer_id="C2_DEPENDENCY_GRAPH", top_k=1),
|
||||
],
|
||||
filters=SimpleNamespace(path_scope=[], test_policy="EXCLUDE"),
|
||||
)
|
||||
|
||||
|
||||
def _constraints() -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
include_globs=["src/**"],
|
||||
exclude_globs=["tests/**"],
|
||||
prefer_globs=["tests/**"],
|
||||
test_file_globs=[],
|
||||
test_symbol_patterns=[],
|
||||
)
|
||||
|
||||
|
||||
def test_retrieve_applies_per_layer_budget_and_dedupe() -> None:
|
||||
repository = _RecordingRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
|
||||
rows = adapter.retrieve("rag-1", "Explain Context", _spec(), _constraints())
|
||||
|
||||
assert [call["limit"] for call in repository.retrieve_calls[:3]] == [2, 3, 1]
|
||||
assert repository.retrieve_calls[0]["layers"] == ["C1_SYMBOL_CATALOG"]
|
||||
assert repository.retrieve_calls[1]["layers"] == ["C0_SOURCE_CHUNKS"]
|
||||
assert repository.retrieve_calls[2]["layers"] == ["C2_DEPENDENCY_GRAPH"]
|
||||
assert repository.retrieve_calls[0]["prefer_path_prefixes"] == ["tests"]
|
||||
assert len([row for row in rows if row["layer"] == "C1_SYMBOL_CATALOG"]) == 1
|
||||
assert len([row for row in rows if row["layer"] == "C0_SOURCE_CHUNKS"]) == 1
|
||||
assert len([row for row in rows if row["layer"] == "C2_DEPENDENCY_GRAPH"]) == 1
|
||||
|
||||
|
||||
def test_hydrate_resolved_symbol_adds_overlapping_c0_chunk() -> None:
|
||||
repository = _RecordingRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
rag_rows = [
|
||||
{
|
||||
"path": "src/context.py",
|
||||
"content": "class Context",
|
||||
"layer": "C1_SYMBOL_CATALOG",
|
||||
"title": "Context",
|
||||
"span_start": 10,
|
||||
"span_end": 20,
|
||||
"metadata": {"symbol_id": "sym-1", "qname": "Context", "blob_sha": "blob-1"},
|
||||
}
|
||||
]
|
||||
spec = _spec()
|
||||
constraints = _constraints()
|
||||
|
||||
rows = adapter.hydrate_resolved_symbol_sources(
|
||||
rag_session_id="rag-1",
|
||||
base_query="Explain Context",
|
||||
rag_rows=rag_rows,
|
||||
symbol_resolution={
|
||||
"status": "resolved",
|
||||
"resolved_symbol": "Context",
|
||||
"alternatives": ["Context", "Context.set"],
|
||||
},
|
||||
retrieval_spec=spec,
|
||||
retrieval_constraints=constraints,
|
||||
)
|
||||
|
||||
c0_rows = [row for row in rows if row["layer"] == "C0_SOURCE_CHUNKS"]
|
||||
assert len(c0_rows) == 1
|
||||
assert c0_rows[0]["path"] == "src/context.py"
|
||||
|
||||
|
||||
def test_retrieve_exact_files_keeps_exact_scope_and_disables_fallback() -> None:
|
||||
repository = _RecordingRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
|
||||
rows = adapter.retrieve_exact_files(
|
||||
"rag-1",
|
||||
paths=["./src//context.py"],
|
||||
layers=["C0_SOURCE_CHUNKS"],
|
||||
limit=200,
|
||||
query="open file",
|
||||
ranking_profile="code",
|
||||
)
|
||||
report = adapter.consume_retrieval_report()
|
||||
|
||||
assert repository.retrieve_exact_calls[0]["paths"] == ["src/context.py"]
|
||||
assert repository.retrieve_exact_calls[0]["layers"] == ["C0_SOURCE_CHUNKS"]
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["path"] == "src/context.py"
|
||||
assert report is not None
|
||||
assert report["fallback"]["used"] is False
|
||||
assert report["applied"][0]["effective_path_scope"] == ["src/context.py"]
|
||||
|
||||
|
||||
def test_explain_symbol_candidates_use_symbol_query_and_symbol_search_mode() -> None:
|
||||
repository = _RecordingRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
spec = _spec()
|
||||
query_plan = SimpleNamespace(sub_intent="EXPLAIN", symbol_candidates=["Context"], symbol_kind_hint="class")
|
||||
|
||||
rows = adapter.retrieve_with_plan(
|
||||
"rag-1",
|
||||
"Объясни как работает Context",
|
||||
spec,
|
||||
_constraints(),
|
||||
query_plan=query_plan,
|
||||
)
|
||||
report = adapter.consume_retrieval_report()
|
||||
|
||||
assert rows
|
||||
assert repository.retrieve_calls[0]["query_text"] == "Context"
|
||||
assert report is not None
|
||||
assert report["retrieval_mode_by_layer"]["C1_SYMBOL_CATALOG"] == "symbol_search"
|
||||
|
||||
|
||||
def test_explain_symbol_mode_disables_scope_relaxation_fallback() -> None:
|
||||
class _NoHitRepository(_RecordingRepository):
|
||||
def retrieve(self, rag_session_id: str, query_embedding: list[float], **kwargs) -> list[dict]: # noqa: ARG002
|
||||
self.retrieve_calls.append(kwargs)
|
||||
if kwargs.get("path_prefixes") is not None:
|
||||
return []
|
||||
return [{"path": "src/other.py", "layer": "C1_SYMBOL_CATALOG", "title": "Other", "metadata": {"kind": "class"}}]
|
||||
|
||||
repository = _NoHitRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
spec = SimpleNamespace(
|
||||
layer_queries=[_LayerQuery(layer_id="C1_SYMBOL_CATALOG", top_k=2)],
|
||||
filters=SimpleNamespace(path_scope=["src/context.py"], test_policy="EXCLUDE"),
|
||||
rerank_profile="code",
|
||||
)
|
||||
query_plan = SimpleNamespace(sub_intent="EXPLAIN", symbol_candidates=["Context"], symbol_kind_hint="class")
|
||||
|
||||
rows = adapter.retrieve_with_plan("rag-1", "Explain Context", spec, _constraints(), query_plan=query_plan)
|
||||
report = adapter.consume_retrieval_report()
|
||||
|
||||
assert rows == []
|
||||
assert len(repository.retrieve_calls) == 1
|
||||
assert repository.retrieve_calls[0]["path_prefixes"] == ["src/context.py"]
|
||||
assert report is not None
|
||||
|
||||
|
||||
def test_explain_symbol_mode_uses_symbol_query_for_non_c1_layers_and_filters_c4_noise() -> None:
|
||||
repository = _RecordingRepository()
|
||||
adapter = RagDbAdapter(repository=repository, dim_resolver=_FakeDimResolver())
|
||||
spec = SimpleNamespace(
|
||||
layer_queries=[
|
||||
_LayerQuery(layer_id="C1_SYMBOL_CATALOG", top_k=2),
|
||||
_LayerQuery(layer_id="C4_SEMANTIC_ROLES", top_k=4),
|
||||
],
|
||||
filters=SimpleNamespace(path_scope=[], test_policy="EXCLUDE"),
|
||||
rerank_profile="code",
|
||||
)
|
||||
query_plan = SimpleNamespace(sub_intent="EXPLAIN", symbol_candidates=["Context"], symbol_kind_hint="class")
|
||||
|
||||
rows = adapter.retrieve_with_plan("rag-1", "Объясни как работает Context", spec, _constraints(), query_plan=query_plan)
|
||||
report = adapter.consume_retrieval_report()
|
||||
|
||||
c4_calls = [call for call in repository.retrieve_calls if call["layers"] == ["C4_SEMANTIC_ROLES"]]
|
||||
c4_rows = [row for row in rows if row["layer"] == "C4_SEMANTIC_ROLES"]
|
||||
|
||||
assert c4_calls
|
||||
assert c4_calls[0]["query_text"] == "Context"
|
||||
assert len(c4_rows) == 1
|
||||
assert c4_rows[0]["title"] == "Context"
|
||||
assert report is not None
|
||||
assert report["fallback"]["used"] is False
|
||||
@@ -0,0 +1,18 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from tests.pipeline_setup.utils.rag_indexer import LocalRepoFileCollector
|
||||
|
||||
|
||||
def test_local_repo_file_collector_skips_hidden_and_pycache_paths(tmp_path) -> None:
|
||||
(tmp_path / ".env").write_text("A=1", encoding="utf-8")
|
||||
(tmp_path / ".venv").mkdir()
|
||||
(tmp_path / ".venv" / "lib.py").write_text("x=1", encoding="utf-8")
|
||||
(tmp_path / "src").mkdir()
|
||||
(tmp_path / "src" / "__pycache__").mkdir()
|
||||
(tmp_path / "src" / "__pycache__" / "cache.py").write_text("x=2", encoding="utf-8")
|
||||
(tmp_path / "src" / ".hidden.py").write_text("x=3", encoding="utf-8")
|
||||
(tmp_path / "src" / "main.py").write_text("def main():\n return 1\n", encoding="utf-8")
|
||||
|
||||
files = LocalRepoFileCollector(tmp_path).collect()
|
||||
|
||||
assert [item["path"] for item in files] == ["src/main.py"]
|
||||
@@ -0,0 +1,140 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.modules.rag.intent_router_v2 import ConversationState, IntentRouterV2
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.phrases_loader import PhraseCatalogLoader
|
||||
from tests.unit_tests.rag.intent_router_testkit import repo_context
|
||||
|
||||
|
||||
def _by_id(case_id: str):
|
||||
cases = PhraseCatalogLoader().load(Path(__file__).resolve().parent / "fixtures" / "phrases.yaml")
|
||||
for case in cases:
|
||||
if case.case_id == case_id:
|
||||
return case
|
||||
raise AssertionError(f"case {case_id} not found")
|
||||
|
||||
|
||||
def test_show_file_phrase_maps_to_open_file() -> None:
|
||||
case = _by_id("code-open-abstract-task")
|
||||
result = IntentRouterV2().route(case.text, ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "OPEN_FILE"
|
||||
assert [item.layer_id for item in result.retrieval_spec.layer_queries] == ["C0_SOURCE_CHUNKS"]
|
||||
assert result.query_plan.path_hints
|
||||
assert result.query_plan.keyword_hints == []
|
||||
assert result.query_plan.symbol_candidates == []
|
||||
assert result.retrieval_constraints.fuzzy_symbol_search.enabled is False
|
||||
|
||||
|
||||
def test_find_tests_prefers_test_scope() -> None:
|
||||
case = _by_id("code-find-tests-for-context")
|
||||
result = IntentRouterV2().route(case.text, ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "FIND_TESTS"
|
||||
assert "tests/**" in result.retrieval_constraints.prefer_globs
|
||||
layers = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert "C3_ENTRYPOINTS" not in layers
|
||||
|
||||
|
||||
def test_explain_function_has_symbol_kind_hint() -> None:
|
||||
case = _by_id("code-explain-handle-errors")
|
||||
result = IntentRouterV2().route(case.text, ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.symbol_kind_hint == "function"
|
||||
assert "tests/**" in result.retrieval_constraints.exclude_globs
|
||||
assert result.symbol_resolution.status == "pending"
|
||||
|
||||
|
||||
def test_followup_marker_sets_followup_likely_mode() -> None:
|
||||
router = IntentRouterV2()
|
||||
first = router.route("Открой файл src/mail_order_bot/context.py", ConversationState(), repo_context())
|
||||
state = ConversationState().advance(first)
|
||||
|
||||
second = router.route("Теперь объясни функцию handle_errors", state, repo_context())
|
||||
|
||||
assert second.conversation_mode == "FOLLOWUP_LIKELY"
|
||||
assert second.query_plan.symbol_candidates == ["handle_errors"]
|
||||
|
||||
|
||||
def test_docs_profile_uses_docs_constraints_and_disables_symbol_resolution() -> None:
|
||||
case = _by_id("docs-about-readme-deploy")
|
||||
result = IntentRouterV2().route(case.text, ConversationState(), repo_context())
|
||||
|
||||
assert result.retrieval_profile == "docs"
|
||||
assert result.symbol_resolution.status == "not_requested"
|
||||
assert result.query_plan.symbol_candidates == []
|
||||
assert result.query_plan.symbol_kind_hint == "unknown"
|
||||
assert "README_DEPLOY.md" in result.retrieval_constraints.include_globs
|
||||
assert "src/**" not in result.retrieval_constraints.include_globs
|
||||
assert any(item in result.query_plan.keyword_hints for item in ["deploy", "deployment", "docker", "compose"])
|
||||
|
||||
|
||||
def test_docs_path_scope_prefers_scoped_layers() -> None:
|
||||
case = _by_id("docs-about-readme-deploy")
|
||||
result = IntentRouterV2().route(case.text, ConversationState(), repo_context())
|
||||
|
||||
layers = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert layers == ["D3_SECTION_INDEX", "D2_FACT_INDEX"]
|
||||
|
||||
|
||||
def test_trace_flow_phrase_maps_to_trace_flow_sub_intent() -> None:
|
||||
result = IntentRouterV2().route("Покажи поток данных для Context.data", ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "TRACE_FLOW"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert "C2_DEPENDENCY_GRAPH" in layer_ids
|
||||
assert "C3_ENTRYPOINTS" in layer_ids
|
||||
assert result.evidence_policy.require_flow is True
|
||||
|
||||
|
||||
def test_explain_sub_intent_keeps_entrypoints_for_execution_trace_retrieval() -> None:
|
||||
result = IntentRouterV2().route("Объясни как работает Context", ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "EXPLAIN"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert "C3_ENTRYPOINTS" in layer_ids
|
||||
assert "C4_SEMANTIC_ROLES" in layer_ids
|
||||
|
||||
|
||||
def test_architecture_phrase_uses_architecture_sub_intent_and_semantic_roles_layer() -> None:
|
||||
result = IntentRouterV2().route("Какие сервисы и обработчики есть в архитектуре?", ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "ARCHITECTURE"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert layer_ids[0] == "C4_SEMANTIC_ROLES"
|
||||
assert "C2_DEPENDENCY_GRAPH" in layer_ids
|
||||
assert "C3_ENTRYPOINTS" in layer_ids
|
||||
|
||||
|
||||
def test_endpoint_phrase_maps_to_find_entrypoints() -> None:
|
||||
result = IntentRouterV2().route("Где health endpoint?", ConversationState(), repo_context())
|
||||
|
||||
assert result.query_plan.sub_intent == "FIND_ENTRYPOINTS"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert layer_ids == ["C3_ENTRYPOINTS", "C0_SOURCE_CHUNKS"]
|
||||
|
||||
|
||||
def test_request_flow_phrase_prefers_trace_flow_over_explain() -> None:
|
||||
result = IntentRouterV2().route(
|
||||
"Покажи как проходит запрос /health в HttpControlAppFactory",
|
||||
ConversationState(),
|
||||
repo_context(),
|
||||
)
|
||||
|
||||
assert result.query_plan.sub_intent == "TRACE_FLOW"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert "C2_DEPENDENCY_GRAPH" in layer_ids
|
||||
assert "C3_ENTRYPOINTS" in layer_ids
|
||||
|
||||
|
||||
def test_flow_phrase_with_start_prefers_trace_flow_over_entrypoints() -> None:
|
||||
result = IntentRouterV2().route(
|
||||
"Покажи поток выполнения при запуске RuntimeManager",
|
||||
ConversationState(),
|
||||
repo_context(),
|
||||
)
|
||||
|
||||
assert result.query_plan.sub_intent == "TRACE_FLOW"
|
||||
layer_ids = [item.layer_id for item in result.retrieval_spec.layer_queries]
|
||||
assert "C2_DEPENDENCY_GRAPH" in layer_ids
|
||||
@@ -0,0 +1,250 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.models import PhraseCase, PipelineResult
|
||||
from tests.pipeline_setup.suite_02_pipeline.pipeline_intent_rag.helpers.runtime import PipelineRuntime
|
||||
|
||||
|
||||
class _FakeAnswerer:
|
||||
def build_prompt_payload(
|
||||
self,
|
||||
query: str,
|
||||
rag_rows: list[dict],
|
||||
*,
|
||||
prompt_template: dict | None = None,
|
||||
sub_intent: str | None = None,
|
||||
) -> dict:
|
||||
return {
|
||||
"system_prompt": "sys",
|
||||
"user_prompt": f"q={query} rows={len(rag_rows)} sub={sub_intent}",
|
||||
"diagnostics": {
|
||||
"prompt_stats": {"tokens_in_estimate": 10, "evidence_rows": len(rag_rows), "evidence_chars": 100},
|
||||
"evidence_summary": [],
|
||||
"prompt_template_id": str((prompt_template or {}).get("template_id") or "t"),
|
||||
"system_prompt_version": "v1",
|
||||
},
|
||||
}
|
||||
|
||||
def answer_from_payload(self, payload: dict) -> str:
|
||||
return "ok"
|
||||
|
||||
|
||||
class _ShouldNotBeCalledAnswerer:
|
||||
def __init__(self) -> None:
|
||||
raise AssertionError("LLM should not be instantiated for missing OPEN_FILE")
|
||||
|
||||
|
||||
class _FakeModel:
|
||||
def __init__(self, **payload) -> None:
|
||||
self.__dict__.update(payload)
|
||||
|
||||
def model_dump(self) -> dict:
|
||||
return dict(self.__dict__)
|
||||
|
||||
|
||||
def _fake_executor_result(
|
||||
*,
|
||||
query: str,
|
||||
sub_intent: str,
|
||||
rag_rows: list[dict],
|
||||
final_answer: str,
|
||||
answer_mode: str,
|
||||
llm_used: bool,
|
||||
validation_reasons: list[str],
|
||||
path_scope: list[str] | None = None,
|
||||
symbol_candidates: list[str] | None = None,
|
||||
symbol_resolution: dict | None = None,
|
||||
layers: list[str] | None = None,
|
||||
) -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
final_answer=final_answer,
|
||||
answer_mode=answer_mode,
|
||||
llm_used=llm_used,
|
||||
validation=SimpleNamespace(reasons=validation_reasons),
|
||||
retrieval_result=SimpleNamespace(raw_rows=rag_rows),
|
||||
router_result=SimpleNamespace(
|
||||
query_plan=SimpleNamespace(symbol_candidates=symbol_candidates or []),
|
||||
symbol_resolution=_FakeModel(**(symbol_resolution or {"status": "not_requested", "resolved_symbol": None, "alternatives": [], "confidence": 0.0})),
|
||||
),
|
||||
diagnostics=SimpleNamespace(
|
||||
router_result={
|
||||
"intent": "CODE_QA",
|
||||
"graph_id": "CodeQAGraph",
|
||||
"conversation_mode": "START",
|
||||
"sub_intent": sub_intent,
|
||||
},
|
||||
retrieval_request={
|
||||
"query": query,
|
||||
"path_scope": path_scope or [],
|
||||
"requested_layers": layers or [],
|
||||
},
|
||||
timings_ms={},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_full_chain_writes_single_record_with_steps(monkeypatch) -> None:
|
||||
test_root = Path(__file__).resolve().parent
|
||||
runtime = PipelineRuntime(mode="full_chain", test_name="test_runtime_full_chain_output", test_root=test_root)
|
||||
case = PhraseCase(case_id="c1", text="Открой файл src/mail_order_bot/context.py", expected_intent="CODE_QA")
|
||||
rag_rows = [
|
||||
{
|
||||
"path": "src/mail_order_bot/context.py",
|
||||
"layer": "C0_SOURCE_CHUNKS",
|
||||
"title": "src/mail_order_bot/context.py:1-10",
|
||||
"content": "class Context: ...",
|
||||
"metadata": {},
|
||||
}
|
||||
]
|
||||
writes: list[dict] = []
|
||||
|
||||
monkeypatch.setattr(runtime._writer, "write_record", lambda payload: writes.append(payload))
|
||||
monkeypatch.setattr(
|
||||
runtime,
|
||||
"_executor_instance",
|
||||
lambda: SimpleNamespace(
|
||||
execute=lambda **_: _fake_executor_result(
|
||||
query=case.text,
|
||||
sub_intent="OPEN_FILE",
|
||||
rag_rows=rag_rows,
|
||||
final_answer="ok",
|
||||
answer_mode="normal",
|
||||
llm_used=True,
|
||||
validation_reasons=[],
|
||||
path_scope=["src/mail_order_bot/context.py"],
|
||||
layers=["C0_SOURCE_CHUNKS"],
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
runtime.run_full_chain(case)
|
||||
|
||||
assert len(writes) == 1
|
||||
payload = writes[0]
|
||||
assert payload["mode"] == "full_chain"
|
||||
assert payload["summary"]["llm"]["answer_status"] == "answered"
|
||||
assert payload["run_info"]["case_id"] == case.case_id
|
||||
assert payload["input_request"]["text"] == case.text
|
||||
assert [item["step"] for item in payload["steps"]] == ["intent_router", "retrieval", "llm_answer"]
|
||||
|
||||
|
||||
def test_full_chain_short_circuits_missing_open_file(monkeypatch) -> None:
|
||||
test_root = Path(__file__).resolve().parent
|
||||
runtime = PipelineRuntime(mode="full_chain", test_name="test_runtime_full_chain_output", test_root=test_root)
|
||||
case = PhraseCase(case_id="missing", text="Открой файл src/app_runtime/core/missing_runtime.py", expected_intent="CODE_QA")
|
||||
writes: list[dict] = []
|
||||
|
||||
monkeypatch.setattr(runtime._writer, "write_record", lambda payload: writes.append(payload))
|
||||
monkeypatch.setattr(
|
||||
runtime,
|
||||
"_executor_instance",
|
||||
lambda: SimpleNamespace(
|
||||
execute=lambda **_: _fake_executor_result(
|
||||
query=case.text,
|
||||
sub_intent="OPEN_FILE",
|
||||
rag_rows=[],
|
||||
final_answer="Файл src/app_runtime/core/missing_runtime.py не найден.",
|
||||
answer_mode="degraded",
|
||||
llm_used=False,
|
||||
validation_reasons=["file_not_found"],
|
||||
path_scope=["src/app_runtime/core/missing_runtime.py"],
|
||||
layers=["C0_SOURCE_CHUNKS"],
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
result = runtime.run_full_chain(case)
|
||||
|
||||
assert result.llm_answer == "Файл src/app_runtime/core/missing_runtime.py не найден."
|
||||
assert len(writes) == 1
|
||||
payload = writes[0]
|
||||
assert payload["summary"]["llm"]["answer_status"] == "degraded"
|
||||
assert payload["diagnostics"]["answer_policy"]["failure_reason"] == "file_not_found"
|
||||
|
||||
|
||||
def test_full_chain_short_circuits_missing_explain_symbol(monkeypatch) -> None:
|
||||
test_root = Path(__file__).resolve().parent
|
||||
runtime = PipelineRuntime(mode="full_chain", test_name="test_runtime_full_chain_output", test_root=test_root)
|
||||
case = PhraseCase(case_id="missing-symbol", text="Объясни класс RuntimeFactoryManager", expected_intent="CODE_QA")
|
||||
writes: list[dict] = []
|
||||
|
||||
monkeypatch.setattr(runtime._writer, "write_record", lambda payload: writes.append(payload))
|
||||
monkeypatch.setattr(
|
||||
runtime,
|
||||
"_executor_instance",
|
||||
lambda: SimpleNamespace(
|
||||
execute=lambda **_: _fake_executor_result(
|
||||
query=case.text,
|
||||
sub_intent="EXPLAIN",
|
||||
rag_rows=[],
|
||||
final_answer="Сущность RuntimeFactoryManager не найдена. Ближайшие варианты: RuntimeManager, WorkflowRuntimeFactory.",
|
||||
answer_mode="degraded",
|
||||
llm_used=False,
|
||||
validation_reasons=["symbol_not_found"],
|
||||
symbol_candidates=["RuntimeFactoryManager"],
|
||||
symbol_resolution={
|
||||
"status": "not_found",
|
||||
"resolved_symbol": None,
|
||||
"alternatives": ["RuntimeManager", "WorkflowRuntimeFactory"],
|
||||
"confidence": 0.0,
|
||||
},
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
result = runtime.run_full_chain(case)
|
||||
|
||||
assert result.llm_answer.startswith("Сущность RuntimeFactoryManager не найдена")
|
||||
assert "исходя из названия" not in result.llm_answer
|
||||
assert len(writes) == 1
|
||||
payload = writes[0]
|
||||
assert payload["summary"]["llm"]["answer_status"] == "degraded"
|
||||
assert payload["diagnostics"]["answer_policy"]["failure_reason"] == "symbol_not_found"
|
||||
|
||||
|
||||
def test_full_chain_synthesizes_confirmed_http_entrypoints(monkeypatch) -> None:
|
||||
test_root = Path(__file__).resolve().parent
|
||||
runtime = PipelineRuntime(mode="full_chain", test_name="test_runtime_full_chain_output", test_root=test_root)
|
||||
case = PhraseCase(case_id="entrypoints", text="Где health endpoint?", expected_intent="CODE_QA")
|
||||
rag_rows = [
|
||||
{
|
||||
"layer": "C3_ENTRYPOINTS",
|
||||
"path": "src/app_runtime/control/http_app.py",
|
||||
"title": "GET /health declared in HttpControlAppFactory",
|
||||
"content": "",
|
||||
"metadata": {
|
||||
"http_method": "GET",
|
||||
"route_path": "/health",
|
||||
"declaring_symbol": "HttpControlAppFactory",
|
||||
"handler_symbol": "HttpControlAppFactory.create.health",
|
||||
},
|
||||
}
|
||||
]
|
||||
writes: list[dict] = []
|
||||
|
||||
monkeypatch.setattr(runtime._writer, "write_record", lambda payload: writes.append(payload))
|
||||
monkeypatch.setattr(
|
||||
runtime,
|
||||
"_executor_instance",
|
||||
lambda: SimpleNamespace(
|
||||
execute=lambda **_: _fake_executor_result(
|
||||
query=case.text,
|
||||
sub_intent="FIND_ENTRYPOINTS",
|
||||
rag_rows=rag_rows,
|
||||
final_answer="GET /health — объявлен в HttpControlAppFactory, обрабатывается health",
|
||||
answer_mode="normal",
|
||||
llm_used=True,
|
||||
validation_reasons=[],
|
||||
layers=["C3_ENTRYPOINTS"],
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
result = runtime.run_full_chain(case)
|
||||
|
||||
assert result.llm_answer == "GET /health — объявлен в HttpControlAppFactory, обрабатывается health"
|
||||
assert len(writes) == 1
|
||||
assert writes[0]["summary"]["llm"]["answer_status"] == "answered"
|
||||
Reference in New Issue
Block a user